Also – one good heuristic is to look at what experts in a field think. According to Muller and Bostrom (2014), a sample of the top 100 most-cited authors in AI ascribed a > 70% probability to AI within a century, a 50% chance of superintelligence conditional on human-level, and a 10% chance of existential catastrophe conditional on human level AI.
Well, there’s some pretty heavy selection bias for all the “mean” probabilities in the linked study that cause it to skew from the commentator’s assigned probablility of one in a million, namely that AI researchers are intelligent enough not to assign “one in a million” probabilties to things. If they weren’t that would seriously affect the average and make it significantly closer to his. Might want to take that into account.
Well, there’s some pretty heavy selection bias for all the “mean” probabilities in the linked study that cause it to skew from the commentator’s assigned probablility of one in a million, namely that AI researchers are intelligent enough not to assign “one in a million” probabilties to things. If they weren’t that would seriously affect the average and make it significantly closer to his. Might want to take that into account.