what if what you discover is not a piece of technology, but a piece of prediction, like Anthony said? What if you discover that it seems quite likely, based on the aggregate opinion of a bunch of skilled predictors, that artificial general human intelligence will be possible within 10 years? Well that, yeah, that has some profound implications for the world, for policy, for business, for military. There’s no denying that. I feel sometimes there’s a little bit of an instinct to kind of pretend like no one’s going to notice that AGI is really important. I don’t think that’s the case.
I had friends in the 2010 vicinity, who thought, surely no one in government will recognize the importance of superintelligence in the next decade. I was almost convinced. I had a little more faith than my friends, so I would have won some bets, but I still was surprised to see Barack Obama talking about superintelligence on an interview. I think the first thing is not to underestimate the possibility that, if you’ve made this prediction, maybe somebody else is about to make it, too.
That said, if you’re Metaculus, maybe you just know who’s running prediction markets, who is studying good prediction aggregation systems, and you just know no one’s putting in the effort, and you really might know that you’re the only people on earth who have really made this prediction, or maybe you and only a few other think tanks have managed to actually come up with a good prediction about when superintelligent AI will be produced, and, moreover, that it’s soon. If you discovered that, I would tell you the same thing I would tell anyone who discovers a potentially dangerous idea, which is not to write a blog post about it right away.
I would say, find three close, trusted individuals that you think reason well about human extinction risk, and ask them to think about the consequences and who to tell next. Make sure you’re fair-minded about it. Make sure that you don’t underestimate the intelligence of other people and assume that they’ll never make this prediction, but … [...]
Then do a rollout procedure. In software engineering, you developed a new feature for your software, but it could crash the whole network. It could wreck a bunch of user experiences, so you just give it to a few users and see what they think, and you slowly roll it out. I think a slow rollout procedure is the same thing you should do with any dangerous idea, any potentially dangerous idea. You might not even know the idea is dangerous. You may have developed something that only seems plausibly likely to be a civilizational scale threat, but if you zoom out and look at the world, and you imagine all the humans coming up with ideas that could be civilizational scale threats.
Maybe they’re a piece of technology, maybe they’re dangerous predictions, but no particular prediction or technology is likely to be a threat, so no one in particular decides to be careful with their idea, and whoever actually produces the dangerous idea is no more careful than anyone else, and they release their idea, and it falls into the wrong hands or it gets implemented in a dangerous way by mistake. Maybe someone accidentally builds Skynet. Somebody accidentally releases replicable plans for a cheap nuclear weapon.
If you zoom out, you don’t want everyone to just share everything right away, and you want there to be some threshold of just a little worry that’s just enough to have you ask your friends to think about it first. If you’ve got something that you think is 1% likely to pose an extinction threat, that seems like a small probability, and if you’ve done calibration training, you’ll realize that that’s supposed to feel very unlikely. Nonetheless, if 100 people have a 1% chance of causing human extinction, well someone probably has a good chance of doing it.
If you just think you’ve got a small chance of causing human extinction, go ahead, be a little bit worried. Tell your friends to be a little bit worried with you for like a day or three. Then expand your circle a little bit. See if they can see problems with the idea, see dangers with the idea, and slowly expand, roll out the idea into an expanding circle of responsible people until such time as it becomes clear that the idea is not dangerous, or you manage to figure out in what way it’s dangerous and what to do about it, because it’s quite hard to figure out something as complicated as how to manage a human extinction risk all by yourself or even by a team of three or maybe even ten people. You have to expand your circle of trust, but, at the same time, you can do it methodically like a software rollout, until you come up with a good plan for managing it. As for what the plan will be, I don’t know. That’s why I need you guys to do your slow rollout and figure it out.
Here are some relevant thoughts from Andrew Critch on a FLI podcast episode I just heard (though it was released in 2017):