This is neither a good operationalization of “superintelligence” nor a crux for most models of doom.
Is it not a crux for “classic ai doom scenarios”?
I agree it’s not a crux for what should be currently highly rated models of doom, that’s in large part why I argue this, to remove mind share from the old scenarios.
If that power is instead superintelligence (that doesn’t follow humanity),
My implied best available plan for humanity is to create each successive superintelligence with sufficiently fewer resources that it could not takeover despite its mild efficiency advantage at using resources strategically. Thus, you can create and deploy misaligned superintelligence and not end up in the doom scenarios but get to try again. (This does necessitate that we can catch misalignment, which seems likely given the current nature of Ai systems and that we can put ooms more resources into auditing the models than they can put defending themselves. It also necessitates that we change course when we catch misalignment, thus my recommendations for better corporate and national governance).
It seems to me “we only get one try” continues to be a frequently argued for position and I think it’s in practice false (though I’ve seen some tautological définitions for which it’s true but uninformative). This post contibutes to weakening that position.
the gap between the current best intelligence and the newly created one … Is it not a crux for “classic ai doom scenarios”?
It is not. I gestured at the issue here using the “center of gravity” analogy, how it makes the weaker AIs a general resource that doesn’t specifically protect humanity, but can be repurposed by superintelligence just as well for its own ends, once it’s more capable than humanity at wielding it.
Thus, you can create and deploy misaligned superintelligence and not end up in the doom scenarios but get to try again.
There’s a recent post by Yudkowsky on this, though I don’t know if it can be helpful in practice when the issue isn’t clear enough without needing it. That is, regardless of whether you agree with the claim that superintelligence is extinction-level dangerous in this way, even without gaps, or with many superintelligences in the world rather than one, it’s a much more clear-cut claim that this is indeed how the classical AI doom scenarios work, that they don’t go away (according to their intended internal logic) even when you manage to set up a no-gaps kind of situation. It’s a sorites paradox of AI doom, any reversible and controllable step only takes you closer to the irreversible and qualitatively discontinuous eventuality (or at least so say the classical AI doom scenarios).
My guess for how AI progress goes is that at some point, some team gets an AI that starts generalizing sufficiently well, sufficiently far outside of its training distribution, that it can gain mastery of fields like physics, bioengineering, and psychology, to a high enough degree that it more-or-less singlehandedly threatens the entire world
It’s unlikely that a new AI system would be able to “threaten the entire world” based on its mastery of physics etc if it were not substantially smarter than its predecessors. It would not have enough of an edge to take over, in a world already full of AI systems in place with only slightly lesser capabilities. Do you disagree that this scenario doesn’t require a substantial gap?
(Without a substantial gap, an AI system could try to start taking over but would presumably not have enough advantage to never be detected and then be stopped by the existing set of AI systems)
it makes the weaker AIs a general resource that doesn’t specifically protect humanity, but can be repurposed by superintelligence just as well for its own ends, once it’s more capable than humanity at wielding it.
I clarified that the new slightly more intelligent system would be deployed with fewer resources until we’re sure it’s aligned. And it’s only slightly more intelligent than what’s in place, so I don’t see why you’d think it could take over the previous AI systems, who are actively suspicious of it and monitoring it.
This giant historically unprecedented problem has many ordinary-world valid analogies. Like how you can’t determine if someone is trustworthy to handle a billion dollars by seeing how they handle ten dollars, even if it’s in fact the same person and they’re not getting much smarter, because they can think intelligently about whether it’s a good time to steal the money.
Yudkowsky does not engage with the many differences between evaluating AI systems and humans, which in fact make a lot of the problems here quite solvable, in particular under my assumption of no large capability gap. The amount of simulations and tests we can do on AI can allow us to know about them being aligned without hidden motives much better than we can for humans today (but also I wouldn’t lose hope at identifying if a human had ulterior motives given billions of dollars of resources and work to solve that). I think other people have already presented many of these differences in various AI control articles.
(I haven’t read the whole post again in full just to respond to your comment—if there are more important points you think are relevant to my argument here I’ll respond to any you highlight)
Is it not a crux for “classic ai doom scenarios”?
I agree it’s not a crux for what should be currently highly rated models of doom, that’s in large part why I argue this, to remove mind share from the old scenarios.
My implied best available plan for humanity is to create each successive superintelligence with sufficiently fewer resources that it could not takeover despite its mild efficiency advantage at using resources strategically. Thus, you can create and deploy misaligned superintelligence and not end up in the doom scenarios but get to try again. (This does necessitate that we can catch misalignment, which seems likely given the current nature of Ai systems and that we can put ooms more resources into auditing the models than they can put defending themselves. It also necessitates that we change course when we catch misalignment, thus my recommendations for better corporate and national governance).
It seems to me “we only get one try” continues to be a frequently argued for position and I think it’s in practice false (though I’ve seen some tautological définitions for which it’s true but uninformative). This post contibutes to weakening that position.
I broadly agree with your second paragraph
It is not. I gestured at the issue here using the “center of gravity” analogy, how it makes the weaker AIs a general resource that doesn’t specifically protect humanity, but can be repurposed by superintelligence just as well for its own ends, once it’s more capable than humanity at wielding it.
There’s a recent post by Yudkowsky on this, though I don’t know if it can be helpful in practice when the issue isn’t clear enough without needing it. That is, regardless of whether you agree with the claim that superintelligence is extinction-level dangerous in this way, even without gaps, or with many superintelligences in the world rather than one, it’s a much more clear-cut claim that this is indeed how the classical AI doom scenarios work, that they don’t go away (according to their intended internal logic) even when you manage to set up a no-gaps kind of situation. It’s a sorites paradox of AI doom, any reversible and controllable step only takes you closer to the irreversible and qualitatively discontinuous eventuality (or at least so say the classical AI doom scenarios).
In 2022 Soares writes
It’s unlikely that a new AI system would be able to “threaten the entire world” based on its mastery of physics etc if it were not substantially smarter than its predecessors. It would not have enough of an edge to take over, in a world already full of AI systems in place with only slightly lesser capabilities. Do you disagree that this scenario doesn’t require a substantial gap?
(Without a substantial gap, an AI system could try to start taking over but would presumably not have enough advantage to never be detected and then be stopped by the existing set of AI systems)
I clarified that the new slightly more intelligent system would be deployed with fewer resources until we’re sure it’s aligned. And it’s only slightly more intelligent than what’s in place, so I don’t see why you’d think it could take over the previous AI systems, who are actively suspicious of it and monitoring it.
Re Yudkowsky’s post, it notably says
Yudkowsky does not engage with the many differences between evaluating AI systems and humans, which in fact make a lot of the problems here quite solvable, in particular under my assumption of no large capability gap. The amount of simulations and tests we can do on AI can allow us to know about them being aligned without hidden motives much better than we can for humans today (but also I wouldn’t lose hope at identifying if a human had ulterior motives given billions of dollars of resources and work to solve that). I think other people have already presented many of these differences in various AI control articles.
(I haven’t read the whole post again in full just to respond to your comment—if there are more important points you think are relevant to my argument here I’ll respond to any you highlight)