Adam’s whole position here, to me, is rather silly, even if we limit ourselves to use cases where the Twitter poll is being used only to try and extrapolate towards national sentiment.
I agree except with the last part (it’s not silly when thinking about extrapolating to national sentiment). The key is to what extent is it evidence of [insert thing], and of course if you’re interested in learning more, what are the factors that affect the extent to which it is evidence of [insert thing]? In other words, what are you trying to generalize to, and what interesting things are limiting your ability to generalize to [insert other thing]?
Often we are comfy with generalizing from sample to appropriately-defined population (sample of Zvi Twitter noticers to Zvi Twitter noticers), but when we don’t define the scope of our generalization properly, we get uncomfy again (sample of Zvi Twitter followers to US general population). Often we are interested in the limits of generalizability (e.g., this treatment works for men but not women, isn’t that interesting and useful!), unless the those boundaries are trivial (e.g., vasectomies work for men but not women, gosh!) or we already don’t see them as boundaries (e.g., “what if you had changed the wording from ‘YOU in particular’ to ‘YOU specifically’?).
Interestingness is in the eye of the beholder. Concede to Adam for the moment that the boundaries are not interesting because they are well-known limits to generalizability (selection, wording). Then, is it “bad evidence?” Depends on what you’re trying to generalize to (what it is purported to be evidence of)! Adam waves between Twitter polls being “meaningless” and “does not generalize at all” as in worthless for anything at all, which is obviously mostly false (it should at least generalize to Zvi Twitter noticers, though even then it could suffer from self-selection bias like many other polls), vs. “not representative of general views,” which is not silly and is far more debatable (it’s likely “weak” evidence in that Twitter polls can yield biased estimates on some questions [this is the most charitable interpretation of the position]; it’s possibly “bad” evidence if the bias is so severe that the qualitative conclusions will differ egregiously [this is the most accurate interpretation of the position seeing as he literally wanted to differentiate it from weak evidence] - e.g., if I polled lesbian women on how sexually attractive the opposite sex was to infer how sexually attractive the opposite sex is to people generally). So overall, the position is rather silly (low generalizability is not NO generalizability, and selection and wording ARE interesting factors relevant for understanding people), except on the very specific last part, where it’s not silly (possibly bad evidence) but it is also still probably not correct (probably not bad evidence).
I agree except with the last part (it’s not silly when thinking about extrapolating to national sentiment). The key is to what extent is it evidence of [insert thing], and of course if you’re interested in learning more, what are the factors that affect the extent to which it is evidence of [insert thing]? In other words, what are you trying to generalize to, and what interesting things are limiting your ability to generalize to [insert other thing]?
Often we are comfy with generalizing from sample to appropriately-defined population (sample of Zvi Twitter noticers to Zvi Twitter noticers), but when we don’t define the scope of our generalization properly, we get uncomfy again (sample of Zvi Twitter followers to US general population). Often we are interested in the limits of generalizability (e.g., this treatment works for men but not women, isn’t that interesting and useful!), unless the those boundaries are trivial (e.g., vasectomies work for men but not women, gosh!) or we already don’t see them as boundaries (e.g., “what if you had changed the wording from ‘YOU in particular’ to ‘YOU specifically’?).
Interestingness is in the eye of the beholder. Concede to Adam for the moment that the boundaries are not interesting because they are well-known limits to generalizability (selection, wording). Then, is it “bad evidence?” Depends on what you’re trying to generalize to (what it is purported to be evidence of)! Adam waves between Twitter polls being “meaningless” and “does not generalize at all” as in worthless for anything at all, which is obviously mostly false (it should at least generalize to Zvi Twitter noticers, though even then it could suffer from self-selection bias like many other polls), vs. “not representative of general views,” which is not silly and is far more debatable (it’s likely “weak” evidence in that Twitter polls can yield biased estimates on some questions [this is the most charitable interpretation of the position]; it’s possibly “bad” evidence if the bias is so severe that the qualitative conclusions will differ egregiously [this is the most accurate interpretation of the position seeing as he literally wanted to differentiate it from weak evidence] - e.g., if I polled lesbian women on how sexually attractive the opposite sex was to infer how sexually attractive the opposite sex is to people generally). So overall, the position is rather silly (low generalizability is not NO generalizability, and selection and wording ARE interesting factors relevant for understanding people), except on the very specific last part, where it’s not silly (possibly bad evidence) but it is also still probably not correct (probably not bad evidence).