I doubt a few minutes of pondering will provoke any significantly insightful thoughts, but on the off chance that they do here’s what I’ve got:
A major pitfall of most tests is that they can end up examining a wide variety of confounding variables. For example if the test for rationality is based on a written prompt then it selects against those with dyslexia in spite of their rationality. If it’s based on a spoken prompt then it selects for those with similar accents to the test-giver, or against those who had it read to them in a strange way. Ideally since the thing that we’re selecting for is (I assume) practical reasoning skills, we would want the test to have some similarities to real life.
Thus the thought that comes to mind is an escape room which can be set up and run essentially-identically for each participant, whose puzzling elements require you to make Bayesian updates on multiple propositions that you were given an idea of the likelihood at the start. In order to avoid biasing the tests in favor of those with more general knowledge, the propositions would ideally be totally fictitious. It occurs to me that the elements of real-world pressure and communication would bias the test against those prone to anxiety, but given that that’s a common problem when you’re called on to apply your rationality skills in reality I think that may be an acceptable flaw, if no other options are obviously superior.
I doubt a few minutes of pondering will provoke any significantly insightful thoughts, but on the off chance that they do here’s what I’ve got:
A major pitfall of most tests is that they can end up examining a wide variety of confounding variables. For example if the test for rationality is based on a written prompt then it selects against those with dyslexia in spite of their rationality. If it’s based on a spoken prompt then it selects for those with similar accents to the test-giver, or against those who had it read to them in a strange way. Ideally since the thing that we’re selecting for is (I assume) practical reasoning skills, we would want the test to have some similarities to real life.
Thus the thought that comes to mind is an escape room which can be set up and run essentially-identically for each participant, whose puzzling elements require you to make Bayesian updates on multiple propositions that you were given an idea of the likelihood at the start. In order to avoid biasing the tests in favor of those with more general knowledge, the propositions would ideally be totally fictitious. It occurs to me that the elements of real-world pressure and communication would bias the test against those prone to anxiety, but given that that’s a common problem when you’re called on to apply your rationality skills in reality I think that may be an acceptable flaw, if no other options are obviously superior.