I think this is a very interesting thought experiment, and it is probing something real.
However, I think that it is missing something important in how it maps to graduate admissions. Lets say that for one reason or another applications are few and places are many. Instead of accepting the top 10 out of 100 applications you are going to reject the bottom 10, and accept the other 90. This legibility argument suggests that in such a situation the applicants from the prestigious universities will be disadvantaged. (Simply slide the “accepted” line in your bell curve graph to the left). This feature strikes me as not matching the reality I imagine I live in.
I think the missing layer of this is that, as an applicant (to a job or anything) you usually have the option to intentionally make your application less legible in various ways, by omitting information that would disadvantage you. You know that Paige is not going to give a great estimate of your performance, so you don’t ask her. As a result less legible applications are going to be correlated with weaker ones. The assessor should, to an extent, be interpreting absence of good evidence (evidence that greatly narrows the candidates uncertainty in a rightwards direction) as evidence of absence.
[Exampleina knows she is in the weaker half of the class, and she knows that Mrs Paige knows this, so doesn’t pick her as her reference. Instead she goes to Mr Oblivious, whose opinions are very weakly correlated with reality, but who happens to think Exampleina is incredibly gifted. Exampaul can speak a foreign language, and to make this more legible in his application he pays an outside organisation to set him an exam on this language to get himself a certificate he can mention on his application. Unfortunately, Exampaul over-estimates his fluency, does not prepare for the test, and he scores a D-grade in his fluency test. He doesn’t mention the fluency-test he paid for on the application at all, and simply puts “fluent in {language}” on his form without further evidence.]
Very nice point! We had definitely thought about the fact that when slots are large and candidates are few, that would give people from less prestigious/legible backgrounds an advantage. (We were speculating idly whether we could come up with uncontroversial examples...)
But I don’t think we’d thought about the point that people might intentionally manipulate how legible their application is. That’s a very nice point! I’m wondering a bit how to model it. Obviously if the Bayesian selectors know that they’re doing this and exactly how, they’ll try to price it in (“this is illegible” is evidence that it’s from a less-qualified candidate). But I can’t really see how those dynamics play out yet. Will have to think more about it. Thanks!
You’re right that legibility alone isn’t the whole story, but the reason I think Presties would still be advantaged in the many-slots-few-applicants story is that admissions officers also have a higher prior on Prestie quality. The impact of AOs’ favorable prior about Presties is, I think, well acknowledged; the impact of their more precise signals is not, which is why I think this post is onto something important.
I think this is a very interesting thought experiment, and it is probing something real.
However, I think that it is missing something important in how it maps to graduate admissions. Lets say that for one reason or another applications are few and places are many. Instead of accepting the top 10 out of 100 applications you are going to reject the bottom 10, and accept the other 90. This legibility argument suggests that in such a situation the applicants from the prestigious universities will be disadvantaged. (Simply slide the “accepted” line in your bell curve graph to the left). This feature strikes me as not matching the reality I imagine I live in.
I think the missing layer of this is that, as an applicant (to a job or anything) you usually have the option to intentionally make your application less legible in various ways, by omitting information that would disadvantage you. You know that Paige is not going to give a great estimate of your performance, so you don’t ask her. As a result less legible applications are going to be correlated with weaker ones. The assessor should, to an extent, be interpreting absence of good evidence (evidence that greatly narrows the candidates uncertainty in a rightwards direction) as evidence of absence.
[Exampleina knows she is in the weaker half of the class, and she knows that Mrs Paige knows this, so doesn’t pick her as her reference. Instead she goes to Mr Oblivious, whose opinions are very weakly correlated with reality, but who happens to think Exampleina is incredibly gifted.
Exampaul can speak a foreign language, and to make this more legible in his application he pays an outside organisation to set him an exam on this language to get himself a certificate he can mention on his application. Unfortunately, Exampaul over-estimates his fluency, does not prepare for the test, and he scores a D-grade in his fluency test. He doesn’t mention the fluency-test he paid for on the application at all, and simply puts “fluent in {language}” on his form without further evidence.]
Very nice point! We had definitely thought about the fact that when slots are large and candidates are few, that would give people from less prestigious/legible backgrounds an advantage. (We were speculating idly whether we could come up with uncontroversial examples...)
But I don’t think we’d thought about the point that people might intentionally manipulate how legible their application is. That’s a very nice point! I’m wondering a bit how to model it. Obviously if the Bayesian selectors know that they’re doing this and exactly how, they’ll try to price it in (“this is illegible” is evidence that it’s from a less-qualified candidate). But I can’t really see how those dynamics play out yet. Will have to think more about it. Thanks!
You’re right that legibility alone isn’t the whole story, but the reason I think Presties would still be advantaged in the many-slots-few-applicants story is that admissions officers also have a higher prior on Prestie quality. The impact of AOs’ favorable prior about Presties is, I think, well acknowledged; the impact of their more precise signals is not, which is why I think this post is onto something important.