The type of replies you get, and the skills you are testing, would also depend how long the subject is spending on the test. Did you have a particular time limit in mind?
I think timeboxing it to 3 hours or so would be a good standard; maybe a bit more if you’re totally unfamiliar with poker.
I don’t think judging responses would be particularly difficult; even if we don’t know what actually happened for certain, you can still judge whether someone used valid rules of inference to reach a plausible estimate. (Judging well requires rationality skills too, of course—rationalists should be more easily convinced of true propositions than false ones, and be able to distinguish invalid reasoning from valid reasoning.)
Also, I suspect that most strong rationalists would independently converge to the same probability estimate for approximately the same reasons, if they looked into the matter, which could serve as a baseline.
Also, I suspect that most strong rationalists would independently converge to the same probability estimate for approximately the same reasons, if they looked into the matter, which could serve as a baseline.
ROFL. The very setup of the post (it’s controversial and there’s no consensus, even among professionals who’ve spent a lot more than a few hours looking into it) contradicts this. There’s also a bunch of private information and priors (such as “what is the base rate of cheating in high-stakes poker” and “what side payments had been made among participants and crew”) that are very hard to validate. Even if there were a reasonable base-rate, the question of whether this KIND of cheating (alleged access to hold-card camera feed) is comparable to other kinds (soft-playing or signaling a compatriot, acting out of turn for information, more mild angle-shooting).
The fact that it is controversial among non-rationalists does not mean that it would be similarly controversial among (strong) rationalists.
This is probably not worth their time and too expensive to test, but concretely, I predict: if Duncan Sabien, gwern, Zvi (or other people in this general reference class, or people who did well on the Amanda Knox test for the right reasons, etc.) each spent some hours looking into this, I suspect they would reach mostly the same conclusions for mostly the same reasons independently.
Seems difficult to mark answers to this question.
The type of replies you get, and the skills you are testing, would also depend how long the subject is spending on the test. Did you have a particular time limit in mind?
I think timeboxing it to 3 hours or so would be a good standard; maybe a bit more if you’re totally unfamiliar with poker.
I don’t think judging responses would be particularly difficult; even if we don’t know what actually happened for certain, you can still judge whether someone used valid rules of inference to reach a plausible estimate. (Judging well requires rationality skills too, of course—rationalists should be more easily convinced of true propositions than false ones, and be able to distinguish invalid reasoning from valid reasoning.)
Also, I suspect that most strong rationalists would independently converge to the same probability estimate for approximately the same reasons, if they looked into the matter, which could serve as a baseline.
ROFL. The very setup of the post (it’s controversial and there’s no consensus, even among professionals who’ve spent a lot more than a few hours looking into it) contradicts this. There’s also a bunch of private information and priors (such as “what is the base rate of cheating in high-stakes poker” and “what side payments had been made among participants and crew”) that are very hard to validate. Even if there were a reasonable base-rate, the question of whether this KIND of cheating (alleged access to hold-card camera feed) is comparable to other kinds (soft-playing or signaling a compatriot, acting out of turn for information, more mild angle-shooting).
The fact that it is controversial among non-rationalists does not mean that it would be similarly controversial among (strong) rationalists.
This is probably not worth their time and too expensive to test, but concretely, I predict: if Duncan Sabien, gwern, Zvi (or other people in this general reference class, or people who did well on the Amanda Knox test for the right reasons, etc.) each spent some hours looking into this, I suspect they would reach mostly the same conclusions for mostly the same reasons independently.