I think updating against these base rates is the critical thing.
But it’s not really an update. The key difference between optimists and pessimists in this area is the recognition that there are no base rates for something like AGI. We have developed new technologies before, but we have never developed a new species before.
New forms of intelligence and agency are a completely new phenomena. Sonic you wanted to ascribe a base rate of our surviving this with zero previous examples, you’d put it at .5. if you counted all of the previous hominid extinctions as relevant, you’d actually put the base rate much lower.
This really seems like the relevant comparison. Tools don’t kill you, but strange creatures do. AGI will be a creature, not a tool.
I think updating against these base rates is the critical thing.
But it’s not really an update. The key difference between optimists and pessimists in this area is the recognition that there are no base rates for something like AGI. We have developed new technologies before, but we have never developed a new species before.
New forms of intelligence and agency are a completely new phenomena. Sonic you wanted to ascribe a base rate of our surviving this with zero previous examples, you’d put it at .5. if you counted all of the previous hominid extinctions as relevant, you’d actually put the base rate much lower.
This really seems like the relevant comparison. Tools don’t kill you, but strange creatures do. AGI will be a creature, not a tool.