I asked Claude about this, and here are some of the things it came up with for disambiguating the different confounds like inability to think hypothetically:
• “Imagine a world where water is poisonous to humans but juice is free and everywhere. In this world, would it be a good idea to drink water?” (Anyone saying yes is failing to enter the hypothetical.)
• “Suppose scientists discovered that exercise actually makes people less healthy. In that world, should doctors recommend exercise?” (Tests whether people can override a real-world belief when told to.)
• “If you found a magic lamp and the genie could grant any wish with absolutely no catches or side effects, would it be good to wish for all diseases to be cured?” (Shows something about how people process “no catches” stipulations.)
Basically, questions where there’s a clearly right answer conditional on accepting the hypothetical. The “cure all diseases” one could probably still have the Malthusian suspicion of your “everyone lives forever” question, but I think that something like the first two would be good for looking at basic hypothetical reasoning. I don’t know that these specific questions are good, but that’s the style of what I would consider including.
I asked Claude about this, and here are some of the things it came up with for disambiguating the different confounds like inability to think hypothetically:
• “Imagine a world where water is poisonous to humans but juice is free and everywhere. In this world, would it be a good idea to drink water?” (Anyone saying yes is failing to enter the hypothetical.)
• “Suppose scientists discovered that exercise actually makes people less healthy. In that world, should doctors recommend exercise?” (Tests whether people can override a real-world belief when told to.)
• “If you found a magic lamp and the genie could grant any wish with absolutely no catches or side effects, would it be good to wish for all diseases to be cured?” (Shows something about how people process “no catches” stipulations.)
Basically, questions where there’s a clearly right answer conditional on accepting the hypothetical. The “cure all diseases” one could probably still have the Malthusian suspicion of your “everyone lives forever” question, but I think that something like the first two would be good for looking at basic hypothetical reasoning. I don’t know that these specific questions are good, but that’s the style of what I would consider including.