I would honestly be interested to see a detailed writeup with good examples of this “maybe amazing” vs “probably good” distinction.
A subtlety here is that the traits that make a candidate a potential outlier are often very different from the traits that would make them “pretty good,” so improving your filtering process to produce more “pretty good” candidates won’t necessarily increase the rate of finding outliers, and might even decrease it.
Most important point I’d still want to grok is what this “might even decrease it” looks like. What are industry examples of metrics or filtering processes that can differentiate 95th percentile samples from 99.9th percentile samples? And what are some of the qualitative shifts you see between them? I suspect the art of identifying 99.9th percentile samples goes beyond looking for “really, really good” 95th-percentile-ish things.
I would honestly be interested to see a detailed writeup with good examples of this “maybe amazing” vs “probably good” distinction.
Most important point I’d still want to grok is what this “might even decrease it” looks like. What are industry examples of metrics or filtering processes that can differentiate 95th percentile samples from 99.9th percentile samples? And what are some of the qualitative shifts you see between them? I suspect the art of identifying 99.9th percentile samples goes beyond looking for “really, really good” 95th-percentile-ish things.
https://www.lesswrong.com/posts/dC7mP5nSwvpL65Qu5/why-the-tails-come-apart is relevant.