Oddly, I was just reading How to Lie with Statistics the other day, and, well …
One often-cited study from 1998, however, concludes that attending a more selective or elite institution does not translate to an economic advantage for students later on, as measured by their reported income.
… these data might be very poor, for several reasons.
Self-reporting is unlikely to be accurate—people both overstate and understate income for various reasons.
The surveys can only go out to people that can be tracked down—and those who cannot will be disproportionately the poorly-off.
If the response rate is not close to 100%, there may be a strong difference between those who reply and those who do not.
These are just the three effects I can remember that apply to a study of just this kind.
But how do those skew the result? It’s not enough to say that there’s bias; you would have to show that the bias works against the claimed conclusion. So with respect to your points:
1) If you don’t have a reason to believe the bias skews one way or the other, knowledge that it could be higher or lower doesn’t matter.
2) The wealthier you are, the easier you are to track down, so the survey would tend to over-state the average income.
3) True, but again, do you have a reason to believe it goes one way rather than the other? (I admit I will never understand what kind of person actually responds to surveys. For people in my demographic, responding to a pollster is a mark of shame, and my parents, less reliant on cell phones, refuse to answer pollsters as well.)
Oddly, I was just reading How to Lie with Statistics the other day, and, well …
… these data might be very poor, for several reasons.
Self-reporting is unlikely to be accurate—people both overstate and understate income for various reasons.
The surveys can only go out to people that can be tracked down—and those who cannot will be disproportionately the poorly-off.
If the response rate is not close to 100%, there may be a strong difference between those who reply and those who do not.
These are just the three effects I can remember that apply to a study of just this kind.
But how do those skew the result? It’s not enough to say that there’s bias; you would have to show that the bias works against the claimed conclusion. So with respect to your points:
1) If you don’t have a reason to believe the bias skews one way or the other, knowledge that it could be higher or lower doesn’t matter.
2) The wealthier you are, the easier you are to track down, so the survey would tend to over-state the average income.
3) True, but again, do you have a reason to believe it goes one way rather than the other? (I admit I will never understand what kind of person actually responds to surveys. For people in my demographic, responding to a pollster is a mark of shame, and my parents, less reliant on cell phones, refuse to answer pollsters as well.)
If you have reason to believe that there are significant sources of sample bias in a poll, you will have to discount the results severely.