The big difference between AI and these technologies is that we’re worried about adversarial behavior by the AI.
A more direct analogy would be if Wright & co had been worried that airplanes might “decide” to fly safely until humanity had invented jet engines, then “decide” to crash them all at once. Nuclear bombs do have a direct analogy—a Dr. Strangelove-type scenario in which, after developing an armamentarium in a ostensibly carefully-controlled manner, some madman (or a defect in an automated launch system) triggers an all-out nuclear attack and ends the world.
This is the difficulty, I think. Tech developers naturally want to think in terms of a non-adversarial relationship with their technology. Maybe this is more familiar to biologists like myself than to people working in computer science. We’re often working with living things that can multiply, mutate and spread, and which we know don’t have our best interests in mind. If we achieve AGI, it will be a living in silico organism, and we don’t have a good ability to predict what it’s capable of because it will be unprecedented on the earth.
This is, to a certain extent, the drawing out of my true crux of a lot of a lot of my disagreements with the doomer view, especially on LW. I fundamentally suspect that the trajectory of AI has fundamentally been a story of a normal technology, and in particular I see 1 very important adversarial assumption both fail to materialize, which is evidence against it, given that absence of evidence is evidence of absence, and the assumption didn’t even buy us that much evidence, as it still isn’t predictive enough to be used for AI doom.
I’m talking about, of course, the essentially unbounded instrumental goals/powerseeking assumptions.
Indeed, it’s a pretty large crux, in that if I fundamentally believed that the adversarial framework was a useful model for AI safety, I’d be a lot more worried about AI doom today.
To put it another, this was hitting at one of my true rejections of a lot of the frameworks used often by doomers/decelerationists.
The big difference between AI and these technologies is that we’re worried about adversarial behavior by the AI.
A more direct analogy would be if Wright & co had been worried that airplanes might “decide” to fly safely until humanity had invented jet engines, then “decide” to crash them all at once. Nuclear bombs do have a direct analogy—a Dr. Strangelove-type scenario in which, after developing an armamentarium in a ostensibly carefully-controlled manner, some madman (or a defect in an automated launch system) triggers an all-out nuclear attack and ends the world.
This is the difficulty, I think. Tech developers naturally want to think in terms of a non-adversarial relationship with their technology. Maybe this is more familiar to biologists like myself than to people working in computer science. We’re often working with living things that can multiply, mutate and spread, and which we know don’t have our best interests in mind. If we achieve AGI, it will be a living in silico organism, and we don’t have a good ability to predict what it’s capable of because it will be unprecedented on the earth.
This is, to a certain extent, the drawing out of my true crux of a lot of a lot of my disagreements with the doomer view, especially on LW. I fundamentally suspect that the trajectory of AI has fundamentally been a story of a normal technology, and in particular I see 1 very important adversarial assumption both fail to materialize, which is evidence against it, given that absence of evidence is evidence of absence, and the assumption didn’t even buy us that much evidence, as it still isn’t predictive enough to be used for AI doom.
I’m talking about, of course, the essentially unbounded instrumental goals/powerseeking assumptions.
Indeed, it’s a pretty large crux, in that if I fundamentally believed that the adversarial framework was a useful model for AI safety, I’d be a lot more worried about AI doom today.
To put it another, this was hitting at one of my true rejections of a lot of the frameworks used often by doomers/decelerationists.