Consider how Nobel-Prize winning atomic scientist Ernest Rutherford dismissed the possibility of nuclear power as “the merest moonshine” less than a day before Szilard figured out how to produce such power. In 1901, Wilbur Wright told his brother Orville that “man would not fly for fifty years” – two years later, they flew, leading Wilbur to say that “ever since, I have distrusted myself and avoided all predictions”. Astronomer Joseph de Lalande told the French Academy that “it is impossible” to build a hot air balloon and “only a fool would expect such a thing to be realized”; the Montgolfier brothers flew less than a year later.
(11 years later...) These sort of examples prove nothing if you don’t take into account the size of the group from which you are selecting, as you yourself have pointed out. I think this is a bad argument for a good conclusion. Thanks for the great posts!
Well, there’s some pretty heavy selection bias for all the “mean” probabilities in the linked study that cause it to skew from the commentator’s assigned probablility of one in a million, namely that AI researchers are intelligent enough not to assign “one in a million” probabilties to things. If they weren’t that would seriously affect the average and make it significantly closer to his. Might want to take that into account.