People tend to compartmentalize. We need to bear in mind that anything we come up with that involves testing someone when they know they’re being tested can only check how rational they can be if they put their mind to it, not how rational they are when they’re not being tested.
It is possible to test people for one thing, and claim that you are testing them for another thing. E.g. Asch’s experiments wouldn’t have worked if he had told people the truth about what he was testing for. As long as the person doesn’t know they’re being tested for rationality, it should be OK. You could test people for ability to make money, ability to get some task done, etc.
I agree. The only solutions to this that I can see is to either not let students know when they are being tested, or to have a system of continual testing.
They key is probably to test someone without letting them know you are testing them. If I ran a martial arts dojo and wanted to make sure my students were really super badass ninjas, I would give them a convincing looking “test” that included things you would expect to see: strength, speed, form, technique, success in actual matches, etc.
This would have very little weighting in the actual grade, however. The real test would be some sort of surprise fight or fights where the student has no idea that the fight is actually one of the tests. Perhaps he (or she) is followed by the assailant until an opportunity to pick a fight arises.
The main advantage of the surprise test is that it is much hard to game. Imperfect metrics are much more likely to say something meaningful about the student in this surprise situation than if the student knows the test is coming.
When it comes to the rationality dojo, there are numerous normally easy-to-game heuristics that could be used, for example:
how susceptible the student is to group-think
what they do in some sort of strenuous situation (e.g., do they blow up the Huygens?) The situation must seem real to them.
are they willing to bet their beliefs even when no one important will notice?
What others can you guys think of?
edit: notice that lists are not working.
edit 2: never mind, editing seemed to fix them.
I doubt that it would be practical to analyze all of the information and get a single number as a measure of the student’s rationality. At the top of all of these tests would have to be someone whose judgment on matters of rationality can be trusted. This may be the most difficult part
Also note that this form of testing would probably be expensive.
People tend to compartmentalize. We need to bear in mind that anything we come up with that involves testing someone when they know they’re being tested can only check how rational they can be if they put their mind to it, not how rational they are when they’re not being tested.
It is possible to test people for one thing, and claim that you are testing them for another thing. E.g. Asch’s experiments wouldn’t have worked if he had told people the truth about what he was testing for. As long as the person doesn’t know they’re being tested for rationality, it should be OK. You could test people for ability to make money, ability to get some task done, etc.
http://www.overcomingbias.com/2007/12/aschs-conformit.html
I agree. The only solutions to this that I can see is to either not let students know when they are being tested, or to have a system of continual testing.
They key is probably to test someone without letting them know you are testing them. If I ran a martial arts dojo and wanted to make sure my students were really super badass ninjas, I would give them a convincing looking “test” that included things you would expect to see: strength, speed, form, technique, success in actual matches, etc.
This would have very little weighting in the actual grade, however. The real test would be some sort of surprise fight or fights where the student has no idea that the fight is actually one of the tests. Perhaps he (or she) is followed by the assailant until an opportunity to pick a fight arises.
The main advantage of the surprise test is that it is much hard to game. Imperfect metrics are much more likely to say something meaningful about the student in this surprise situation than if the student knows the test is coming.
When it comes to the rationality dojo, there are numerous normally easy-to-game heuristics that could be used, for example:
how susceptible the student is to group-think
what they do in some sort of strenuous situation (e.g., do they blow up the Huygens?) The situation must seem real to them.
are they willing to bet their beliefs even when no one important will notice?
What others can you guys think of?
edit: notice that lists are not working. edit 2: never mind, editing seemed to fix them.
I doubt that it would be practical to analyze all of the information and get a single number as a measure of the student’s rationality. At the top of all of these tests would have to be someone whose judgment on matters of rationality can be trusted. This may be the most difficult part
Also note that this form of testing would probably be expensive.
See artemis fowl and the butler training.
An insurmountable problem?