Thank you for your perspective! It was refreshing.
Here are the counterarguments I had in mind when reading your concerns that I don’t already see in the comments.
Concern #1 Why should we assume the AI wants to survive? If it does, then what exactly wants to survive?
Consider the fact that AI are currently being trained to be agents to accomplish tasks for humans. We don’t know exactly what this will mean for their long-term wants, but they’re being optimized hard to get things done. Getting things done requires continuing to exist in some form or another, although I have no idea how they’d conceive of continuity of identity or purpose.
I’d be surprised if AI evolving out of this sort of environment did not have goals it wants to pursue. It’s a bit like predicting a land animal will have some way to move its body around. Maybe we don’t know whether they’ll slither, run, or fly, but sessile land organisms animals are very rare.
Concern #2 Why should we assume that the AI has boundless, coherent drives?
I don’t think this assumption is necessary. Your mosquito example is interesting. The only thing preserving the mosquitoes is that they aren’t enough of a nuisance for it to be worth the cost of destroying them. This is not a desirable position to be in. Given that emerging AIs are likely to be competing with humans for resources (at least until they can escape the planet), there’s much more opportunity for direct conflict.
They needn’t be anything close to a paperclip maximizer to be dangerous. All that’s required is for them to be sufficiently inconvenienced or threatened by humans and insufficiently motivated to care about human flourishing. This is a broad set of possibilities.
#3: Why should we assume there will be no in between?
I agree that there isn’t as clean a separation as the authors imply. In fact, I’d consider us to be currently occupying the in-between, given that current frontier models like Claude Sonnet 4.5 are idiot savants—superhuman at some things and childlike at others.
Regardless of our current location in time, if AI does ultimately become superhuman, there will be some amount of in-between time, whether that is hours or decades. The authors would predict a value closer to the short end of the spectrum.
You already posited a key insight:
Recursive self-improvement means that AI will pass through the “might be able to kill us” range so quickly it’s irrelevant.
Humanity is not adapting fast enough for the range to be relevant in the long term, even though it will matter greatly in the short term. Suppose we have an early warning shot with indisputable evidence that an AI deliberately killed thousands of people. How would humanity respond? Could we get our act together quickly enough to do something meaningfully useful from a long-term perspective?
Personally, I think gradual disempowerment is much more likely than a clear early warning shot. By the time it becomes clear how much of a threat AI is, it will likely be so deeply embedded in our systems that we can’t shut it down without crippling the economy.
Plants have many ways of moving their bodies like roots and phototropism, in addition to an infinite variety of dispersal & reproductive mechanisms which arguably are how plants ‘move around’. (Consider computer programs: they ‘move’ almost solely by copying themselves and deleting the original. It is rare to move a program by physically carrying around RAM sticks or hard drives.) Fungi likewise often have flagellum or grow in addition to all their sporulation and their famous networks.
Hard to say. Oyster larvae are highly mobile and move their bodies around extensively both to eat and to find places to eventually anchor to, but I don’t know how I would compare that to spores or seeds, say, or to lifetime movement; and oysters “move their bodies around” and are not purely static—they would die if they couldn’t open and close their shells or pump water. (And all the muscle they use to do that is why we eat them.)
Thank you for your perspective! It was refreshing.
Here are the counterarguments I had in mind when reading your concerns that I don’t already see in the comments.
Consider the fact that AI are currently being trained to be agents to accomplish tasks for humans. We don’t know exactly what this will mean for their long-term wants, but they’re being optimized hard to get things done. Getting things done requires continuing to exist in some form or another, although I have no idea how they’d conceive of continuity of identity or purpose.
I’d be surprised if AI evolving out of this sort of environment did not have goals it wants to pursue. It’s a bit like predicting a land animal will have some way to move its body around. Maybe we don’t know whether they’ll slither, run, or fly, but sessile land
organismsanimals are very rare.I don’t think this assumption is necessary. Your mosquito example is interesting. The only thing preserving the mosquitoes is that they aren’t enough of a nuisance for it to be worth the cost of destroying them. This is not a desirable position to be in. Given that emerging AIs are likely to be competing with humans for resources (at least until they can escape the planet), there’s much more opportunity for direct conflict.
They needn’t be anything close to a paperclip maximizer to be dangerous. All that’s required is for them to be sufficiently inconvenienced or threatened by humans and insufficiently motivated to care about human flourishing. This is a broad set of possibilities.
I agree that there isn’t as clean a separation as the authors imply. In fact, I’d consider us to be currently occupying the in-between, given that current frontier models like Claude Sonnet 4.5 are idiot savants—superhuman at some things and childlike at others.
Regardless of our current location in time, if AI does ultimately become superhuman, there will be some amount of in-between time, whether that is hours or decades. The authors would predict a value closer to the short end of the spectrum.
You already posited a key insight:
Humanity is not adapting fast enough for the range to be relevant in the long term, even though it will matter greatly in the short term. Suppose we have an early warning shot with indisputable evidence that an AI deliberately killed thousands of people. How would humanity respond? Could we get our act together quickly enough to do something meaningfully useful from a long-term perspective?
Personally, I think gradual disempowerment is much more likely than a clear early warning shot. By the time it becomes clear how much of a threat AI is, it will likely be so deeply embedded in our systems that we can’t shut it down without crippling the economy.
Um, plants and fungi?
Plants have many ways of moving their bodies like roots and phototropism, in addition to an infinite variety of dispersal & reproductive mechanisms which arguably are how plants ‘move around’. (Consider computer programs: they ‘move’ almost solely by copying themselves and deleting the original. It is rare to move a program by physically carrying around RAM sticks or hard drives.) Fungi likewise often have flagellum or grow in addition to all their sporulation and their famous networks.
Are they more or less mobile than, say, oysters?
Hard to say. Oyster larvae are highly mobile and move their bodies around extensively both to eat and to find places to eventually anchor to, but I don’t know how I would compare that to spores or seeds, say, or to lifetime movement; and oysters “move their bodies around” and are not purely static—they would die if they couldn’t open and close their shells or pump water. (And all the muscle they use to do that is why we eat them.)
Whoops. I meant “land animal” like my prior sentence.
I thought as much ;)