I think this is conflating two different things: how much effort did you spend (e.g. “Made a 5 minute timer and thought seriously about possible flaws or refinements”) and what did you do to empirically test the idea (e.g. “Ran an Randomized Control Trial”). These two are somewhat correlated, hopefully, but it’s possible both to engage in very complex and effortful flights of fancy without any connection to the empirics, and to start with simple and basic actual tests without thinking about the problem too hard or too long.
I think I’d rather see people state the Falsifiability Status of their idea, say, on a scale ranging from trivial to never. For example:
Trivial: most anyone could do it in a few minutes at most
Easy: many people could do it with a modest investment of time
Moderate: amateurs could do it but it would require effort
Difficult: doable by professionals with a budget; amateurs will have huge difficulties
Very hard: probably doable, but requires large teams and a lot of money and effort
Potentially possible: probably doable in the future, but not at the current technology level
I do think that’s a fair point (I certainly don’t want to encourage people to think hard but non-usefully).
The way I attempted to approach that problem was to have the lower/easier things on the scale be optimized for being easy to get started on (I think the difference between “think seriously for 5 minutes” and “have an actual model of why you think this makes sense” and “not have those things” is fairly extreme, and people who aren’t even doing that should do that).
But I also intended to have it quickly escalate to “think about how to empirically test it, and/or actually empirically test it if possible.” (Not sure if this addresses your concern but I edited the post to make that a bit more clear)
I’d split it up a bit differently. “How much effort” versus “What actual reason does anyone else have to agree with this?”. The latter isn’t quite the same as “what empirical testing has it had?” but I think it’s the more important question.
However, “Epistemic effort” as proposed here (1) probably does correlate pretty well with “how much reason to agree?”, (2) also gives information about the separate question “how seriously is this person taking this discussion?” and (3) is probably easier to give an accurate account of than “what actual reason …”.
“What actual reason does anyone else have to agree with this?”
I think this formulation is a bit iffy, since the monkey-mind might easily have a LOT of reasons to agree with something without any of these reasons being connected to correctly reflecting reality.
I think this is conflating two different things: how much effort did you spend (e.g. “Made a 5 minute timer and thought seriously about possible flaws or refinements”) and what did you do to empirically test the idea (e.g. “Ran an Randomized Control Trial”). These two are somewhat correlated, hopefully, but it’s possible both to engage in very complex and effortful flights of fancy without any connection to the empirics, and to start with simple and basic actual tests without thinking about the problem too hard or too long.
I think I’d rather see people state the Falsifiability Status of their idea, say, on a scale ranging from trivial to never. For example:
Trivial: most anyone could do it in a few minutes at most
Easy: many people could do it with a modest investment of time
Moderate: amateurs could do it but it would require effort
Difficult: doable by professionals with a budget; amateurs will have huge difficulties
Very hard: probably doable, but requires large teams and a lot of money and effort
Potentially possible: probably doable in the future, but not at the current technology level
Never: not falsifiable
I do think that’s a fair point (I certainly don’t want to encourage people to think hard but non-usefully).
The way I attempted to approach that problem was to have the lower/easier things on the scale be optimized for being easy to get started on (I think the difference between “think seriously for 5 minutes” and “have an actual model of why you think this makes sense” and “not have those things” is fairly extreme, and people who aren’t even doing that should do that).
But I also intended to have it quickly escalate to “think about how to empirically test it, and/or actually empirically test it if possible.” (Not sure if this addresses your concern but I edited the post to make that a bit more clear)
I’d split it up a bit differently. “How much effort” versus “What actual reason does anyone else have to agree with this?”. The latter isn’t quite the same as “what empirical testing has it had?” but I think it’s the more important question.
However, “Epistemic effort” as proposed here (1) probably does correlate pretty well with “how much reason to agree?”, (2) also gives information about the separate question “how seriously is this person taking this discussion?” and (3) is probably easier to give an accurate account of than “what actual reason …”.
I think this formulation is a bit iffy, since the monkey-mind might easily have a LOT of reasons to agree with something without any of these reasons being connected to correctly reflecting reality.