Here’s my serious claim after giving this an awful lot of thought and study: you are correct about the arguments for doom being either incomplete or bad.
But the arguments for survival are equally incomplete and bad.
It’s like arguing about whether humans would survive driving a speedboat at ninety miles an hour before anyone’s invented the internal combustion engine. There are some decent arguments that they wouldn’t, and some decent argument that their engineering would be up to the challenge. People could’ve debated endlessly.
The correct answer is that humans often survive driving speedboats at ninety miles and hour and sometimes die doing it. It depends on the quality of engineering and the wisdom of the pilot (and how their motivations are shaped by competition;).
So, to make progress on actual good arguments, you have to talk about specifics: what type of AGI we’ll create, who will be in charge of creating it and perhaps directing it, and exactly what strategies we’ll use to ensure it does what we want and that humanity survives. (That is what my work focuses on.)
In the absence of doing detailed reasoning about specific scenarios, you’re basically taking a guess. It’s no wonder people have wildly varying p(doom) estimates from that method.
If it’s a guess, the base rate is key. You’ve said you refuse to change it, but to understand the issue we have to address it. That’s the primary difference between your p(doom) and mine. I’ve spent a ton of time trying to address specific scenarios, but my base rate is still the dominant factor in my total, because we don’t have enough information yet and havent’ modeled specific scenarios well enough yet.
I think we’ll probably survive a decade after the advent of AGI on something like your odds, but I can’t see farther than that, so my long-range p(doom) goes to around .5 with my base rate. (I’m counting permanent dystopias as doom). (incidentally, Paul Christiano’s actual p(doom) is similar—modest in the near term after AGI but rising to 40% as things progress from there).
It’s tough to arrive at a base rate for something that’s never happened before. There is no reference class, which is why debates about the proper reference class are endless. No species has ever designed and created a new entity smarter than itself. So I put it around .5 for sheer lack of evidence either way. I think people using lower base rates are succumbing to believing what feels easy and good- motivated reasoning. There simply are no relevant observations.
A lot of speedboats are built for leisure. You wouldn’t want to build such speedboats if they were too deadly. People who get killed by speedboats aren’t going to be recommending them to others. The government is going to attack speedboat companies that don’t try to reduce speedboat danger. Speedboats tend to be a net expense rather than a convenience or necessity, so there’s a natural repulsive force away from using them.
I didn’t mean to imply that speedboats are a perfect analogy. They’re not. Maybe not even a good one.
My claim was that details matter; we can’t get good p(doom) estimates without considering specific scenarios, including all of the details you mention about the motivations and regulations around use, as well as the engineering approaches, and more.
If your base rate is strongly different from the expert consensus there should be some explainable reason for the difference.
If the reason for the difference is “I thought a lot about it, but I can’t explain the details to you”, I will happily add yours to the list of “bad arguments”.
A good argument should be:
simple
backed up by facts that are either self-evidently true or empirically observable
If you give me a list of “100 things make me nervous”, I can just as easily give you “a list of 100 things that make me optimistic”.
This was the most compelling part of their post for me:
“You are correct about the arguments for doom being either incomplete or bad. But the arguments for survival are equally incomplete and bad.”
And you really don’t seem to have taken it to heart. You’re demanding that doomers provide you with a good argument. Well, I demand that you provide me with a good argument!
More seriously: we need to weigh the doom-evidence and the non-doom-evidence against each other. But you believe that we need to look at the doom-evidence and if it’s not very good, then p(doom) should be low. But that’s wrong—you don’t acknowledge that the non-doom-evidence is also not very good. IOW there’s a ton of uncertainty.
If you give me a list of “100 things make me nervous”, I can just as easily give you “a list of 100 things that make me optimistic”.
Then it would be a lot more logical for your p(doom) to be 0.5 rather than 0.02-0.2!
Feels like this attitude would lead you to neurotically obsessing over tons of things. You ought to have something that strongly distinguishes AI from other concepts before you start worrying about it, considering how infeasible it is to worry about everything conceivable.
Well of course there is something different: The p(doom), as based on the opinions of a lot of people who I consider to be smart. That strongly distinguishes it from just about every other concept.
“People I consider very smart say this is dangerous” seems so cursed, especially in response to people questioning whether it is dangerous. Would be better for you to not participate in the discussion and just leave it to the people who have an actual independently informed opinion.
How many things could reasonably have a p(doom) > 0.01? Not very many. Therefore your worry about me “neurotically obsessing over tons of things” is unfounded. I promise I won’t :) If my post causes you to think that, then I apologize, I have misspoken my argument.
What is the actual argument that there’s ‘not very many’? (Or why do you believe such an argument made somewhere else)
There’s hundreds of asteroids and comets alone that have some probability of hitting the Earth in the next thousand years, how can anyone possibly evaluate ‘p(doom)’ for any of this, let alone every other possible catastrophe?
I was reading the UK National Risk Register earlier today and thinking about this. Notable to me that the top-level disaster severity has a very low cap of ~thousands of casualties, or billions of economic loss. Although it does note in the register that AI is a chronic risk that is being managed under a new framework (that I can’t find precedent for).
Here’s my serious claim after giving this an awful lot of thought and study: you are correct about the arguments for doom being either incomplete or bad.
But the arguments for survival are equally incomplete and bad.
It’s like arguing about whether humans would survive driving a speedboat at ninety miles an hour before anyone’s invented the internal combustion engine. There are some decent arguments that they wouldn’t, and some decent argument that their engineering would be up to the challenge. People could’ve debated endlessly.
The correct answer is that humans often survive driving speedboats at ninety miles and hour and sometimes die doing it. It depends on the quality of engineering and the wisdom of the pilot (and how their motivations are shaped by competition;).
So, to make progress on actual good arguments, you have to talk about specifics: what type of AGI we’ll create, who will be in charge of creating it and perhaps directing it, and exactly what strategies we’ll use to ensure it does what we want and that humanity survives. (That is what my work focuses on.)
In the absence of doing detailed reasoning about specific scenarios, you’re basically taking a guess. It’s no wonder people have wildly varying p(doom) estimates from that method.
If it’s a guess, the base rate is key. You’ve said you refuse to change it, but to understand the issue we have to address it. That’s the primary difference between your p(doom) and mine. I’ve spent a ton of time trying to address specific scenarios, but my base rate is still the dominant factor in my total, because we don’t have enough information yet and havent’ modeled specific scenarios well enough yet.
I think we’ll probably survive a decade after the advent of AGI on something like your odds, but I can’t see farther than that, so my long-range p(doom) goes to around .5 with my base rate. (I’m counting permanent dystopias as doom). (incidentally, Paul Christiano’s actual p(doom) is similar—modest in the near term after AGI but rising to 40% as things progress from there).
It’s tough to arrive at a base rate for something that’s never happened before. There is no reference class, which is why debates about the proper reference class are endless. No species has ever designed and created a new entity smarter than itself. So I put it around .5 for sheer lack of evidence either way. I think people using lower base rates are succumbing to believing what feels easy and good- motivated reasoning. There simply are no relevant observations.
A lot of speedboats are built for leisure. You wouldn’t want to build such speedboats if they were too deadly. People who get killed by speedboats aren’t going to be recommending them to others. The government is going to attack speedboat companies that don’t try to reduce speedboat danger. Speedboats tend to be a net expense rather than a convenience or necessity, so there’s a natural repulsive force away from using them.
I didn’t mean to imply that speedboats are a perfect analogy. They’re not. Maybe not even a good one.
My claim was that details matter; we can’t get good p(doom) estimates without considering specific scenarios, including all of the details you mention about the motivations and regulations around use, as well as the engineering approaches, and more.
If your base rate is strongly different from the expert consensus there should be some explainable reason for the difference.
If the reason for the difference is “I thought a lot about it, but I can’t explain the details to you”, I will happily add yours to the list of “bad arguments”.
A good argument should be:
simple
backed up by facts that are either self-evidently true or empirically observable
If you give me a list of “100 things make me nervous”, I can just as easily give you “a list of 100 things that make me optimistic”.
There’s a lot of problems with linking to manifold and calling it “the expert consensus”!
It’s not the right source. The survey you linked elsewhere would be better.
Even for the survey, it’s unclear whether these are the “right” experts for the question. This at least needs clarification.
It’s not a consensus, this is a median or mean of a pretty wide distribution.
I wouldn’t belabor it, but you’re putting quite a lot of weight on this one point.
This was the most compelling part of their post for me:
“You are correct about the arguments for doom being either incomplete or bad. But the arguments for survival are equally incomplete and bad.”
And you really don’t seem to have taken it to heart. You’re demanding that doomers provide you with a good argument. Well, I demand that you provide me with a good argument!
More seriously: we need to weigh the doom-evidence and the non-doom-evidence against each other. But you believe that we need to look at the doom-evidence and if it’s not very good, then p(doom) should be low. But that’s wrong—you don’t acknowledge that the non-doom-evidence is also not very good. IOW there’s a ton of uncertainty.
Then it would be a lot more logical for your p(doom) to be 0.5 rather than 0.02-0.2!
Feels like this attitude would lead you to neurotically obsessing over tons of things. You ought to have something that strongly distinguishes AI from other concepts before you start worrying about it, considering how infeasible it is to worry about everything conceivable.
Well of course there is something different: The p(doom), as based on the opinions of a lot of people who I consider to be smart. That strongly distinguishes it from just about every other concept.
“People I consider very smart say this is dangerous” seems so cursed, especially in response to people questioning whether it is dangerous. Would be better for you to not participate in the discussion and just leave it to the people who have an actual independently informed opinion.
How many things could reasonably have a p(doom) > 0.01? Not very many. Therefore your worry about me “neurotically obsessing over tons of things” is unfounded. I promise I won’t :) If my post causes you to think that, then I apologize, I have misspoken my argument.
What is the actual argument that there’s ‘not very many’? (Or why do you believe such an argument made somewhere else)
There’s hundreds of asteroids and comets alone that have some probability of hitting the Earth in the next thousand years, how can anyone possibly evaluate ‘p(doom)’ for any of this, let alone every other possible catastrophe?
I was reading the UK National Risk Register earlier today and thinking about this. Notable to me that the top-level disaster severity has a very low cap of ~thousands of casualties, or billions of economic loss. Although it does note in the register that AI is a chronic risk that is being managed under a new framework (that I can’t find precedent for).