Thus, the chance that a female got an 800 on the Math SAT due to luck is higher than the chance that a male got an 800 on the Math SAT due to luck.
Shouldn’t it be possible to estimate the magnitude of this effect by comparing score distributions on tests with differently sized question pools, or write-in versus multiple choice, or which are otherwise more or less susceptible to luck?
You’d need a model of how much luck depends on those factors. Test-retest variability gives a good measure of how much one person’s scores vary from test to test; apparently for the SAT the test-retest standard deviation is about 30 points. (We can’t quite apply this number, since it might not be independent of score, but it’s better than nothing.)
Shouldn’t it be possible to estimate the magnitude of this effect by comparing score distributions on tests with differently sized question pools, or write-in versus multiple choice, or which are otherwise more or less susceptible to luck?
You’d need a model of how much luck depends on those factors. Test-retest variability gives a good measure of how much one person’s scores vary from test to test; apparently for the SAT the test-retest standard deviation is about 30 points. (We can’t quite apply this number, since it might not be independent of score, but it’s better than nothing.)
That’s part of the whole “getting more information” thing.
I think.