The principle here is that if you’re trying to make the result sound as impressive as possible, an unintended consequence is that you’re revealing the upper limit.
Here is the full quote by Phil Price (from Andrew Gelman’s blog):
Many years ago I saw an ad for a running shoe (maybe it was Reebok?) that said something like “At the New York Marathon, three of the five fastest runners were wearing our shoes.” I’m sure I’m not the first or last person to have realized that there’s more information there than it seems at first. For one thing, you can be sure that one of those three runners finished fifth: otherwise the ad would have said “three of the four fastest.” Also, it seems almost certain that the two fastest runners were not wearing the shoes, and indeed it probably wasn’t 1-3 or 2-3 either: “The two fastest” and “two of the three fastest” both seem better than “three of the top five.” The principle here is that if you’re trying to make the result sound as impressive as possible, an unintended consequence is that you’re revealing the upper limit.
In this case I don’t really think that two of the three fasted is better then three of the five fasted. The bigger the group of running the less likely it is that the effect is random.
Most statisticians would agree with you. Unless Reeboks expected market share were between 2⁄3 and 3⁄5 of course. Though I expect that most laymen would have Phil’s Intuition, in any case the general point: That the statement leaks information remains :)
If a single shoe has an expected market share between 2⁄3 or 3⁄5 of top runners that suggest it’s the best running shoe. There isn’t really that much difference between the two.
Prior to carbon plates I would disagree with this. It would suggest the shoe manufacturer has the largest marketing spend. (In a post-carbon-plate world where people are covering over the swoosh so that (not-Nike) sponsored athletes can race in Nike shoes I think it’s pretty clear which shoe is best)
I disagree, if Reebok produced 64% of all shoes in the world and only 3⁄5 of top athletes used them, and this furthermore was the best statistics the marketing department could produce, then it’s strong evidence that they are over hyped.
But I think you understood me as saying something different, words are hard :)
Reebok doesn’t produce 64% of all shoes or a majority of shoes and that’s generally known by shoe buyers.
I think we agree and are talking past each other, my original statement was “Most statisticians would agree with you. Unless...”
So we agree that there is more power in 3⁄5 than 2⁄3, and we happen to have divergent intuitions about what random Joe finds most persuasive, my intuition is rather weak so I would gladly update it towards 3⁄5 sounding more impressive to random people, if you feel strongly about it.
Most likely what the marketing folks have done is gotten a list of top 100 runners in different running disciplines and the reported “the most impressive top X in list Y”,
We both agree that the reported statistic is inflated, which is the major thesis, we simply disagree about how much information can be recovered because we have different “impressiveness sounding heuristics”
This implies that advertisers would be better off if they occasionally violated such assumptions (such as saying “of the top five” when they were in the top four) enough that it weakens the inferences viewers can make, by enough to benefit the advertisers.
Of course, the coordination problem in doing this is hard, but there are several ways around it (and not all of them just involve advertisers directly colluding with each other).
Honestly, since only a small group intersection of logic-savvy and ad-watching people is ever going to notice such a glomarization, it’s not worth it, even if the coordination problems between advertisers was solved.
Maybe this is a better tool for structuring the discourse then it is for firmly deriving the basis of a given statement.
I’m thinking about the reports my kid gives about his football games. If the final count was 10:9 and the game was long, I ask if it were the final goal which took so much time to score; not because I expect it to be exactly so, but rather because this is the streetlight under which I start looking for the keys.
If, OTOH, it ended at 5:3, I ask what he remembers as the most interesting points. And if it ended at 9:1...
After having read 3 sentences, I already find myself disagreeing with this analysis. “X out of 4” is nowhere near as common as “X out of 5“, since humans like nice round numbers, and culturally we consider 5 to be a round number- so even if #5 was not one of the “3 out of 5” who wore the shoes, “3 out of 5” is still the appropriate phrasing to use.
Similarly, “2 out of 3” isn’t as compelling, because it doesn’t tell us that a large number of talented athletes are wearing the shoes, to the same extent as “3 out of 5″
So this analysis is attempting to extract far more information than is actually justified
If Reebok wanted to report a valid statistic they would report something like 11% of the top 100 wears our shoos, I think a much smaller number than top 100 was picked exactly because that was where the effect was the most exaggerated. ChristianKl share your intuition that 3⁄5 sounds more impressive than 2⁄3. I also agree that reporting top 4 would seem even more fishy, though it could be spun as 75% of quarter finalists in knockout tournament sports.
Huh I think the linkpost didn’t fully work
What’s wrong with it?, I am linking to the source material, should i only link if its a 100% copy?