I do think that’s a fair point (I certainly don’t want to encourage people to think hard but non-usefully).
The way I attempted to approach that problem was to have the lower/easier things on the scale be optimized for being easy to get started on (I think the difference between “think seriously for 5 minutes” and “have an actual model of why you think this makes sense” and “not have those things” is fairly extreme, and people who aren’t even doing that should do that).
But I also intended to have it quickly escalate to “think about how to empirically test it, and/or actually empirically test it if possible.” (Not sure if this addresses your concern but I edited the post to make that a bit more clear)
I do think that’s a fair point (I certainly don’t want to encourage people to think hard but non-usefully).
The way I attempted to approach that problem was to have the lower/easier things on the scale be optimized for being easy to get started on (I think the difference between “think seriously for 5 minutes” and “have an actual model of why you think this makes sense” and “not have those things” is fairly extreme, and people who aren’t even doing that should do that).
But I also intended to have it quickly escalate to “think about how to empirically test it, and/or actually empirically test it if possible.” (Not sure if this addresses your concern but I edited the post to make that a bit more clear)