One has to assume by default the jaggedness of capabilities will persist
I’m talking about systems that are much more capable than humans in ~all relevant dimensions. (They might also be wildly, wildly more capable in some particular dimensions.) If you don’t think such systems are plausible soon-ish (e.g. next 10-20 years), that might be driving the disagreement.
Why wouldn’t the same “sloppiness” that plagues the hypothetical future AI safety research equally plague any sufficiently complex real-world takeover plan?
It depends on whether the cause of the sloppiness is a competence problem or if it’s due to misalignment (the AI totally could do a good job, but it’s trying to do something else). What we have today is somewhere in between.
For the takeover threat model, I’m imagining a world where AIs are capable of doing a good job on extremely sophisticated real world tasks like “fully autonomously run all the manufactoring and R&D for a robot army that greatly outstrips early-2026 conventional militaries”. So, there is an important sense in which these future AIs aren’t sloppy at all. By this point, I also expect AIs won’t qualitatively feel sloppy, though it may in practice be very hard to elicit good conceptual safety research from them due to verification difficulties (while you can verify an apparently functioning robot army etc). I think the slop concern and the “the AIs literally take over due to reward/fitness/apparant-success seeking” are reasonable separate (and hit at differnet points in the capabilities progression) though they may have the same underlying causes. For the same reason these AIs can do a good job building and operating a military, I think they will have the ability to takeover, though the story is complicated because it might be possible to setup the incentives/reinforcement for some of the AIs to whistleblow (but also it might be hard to setup the incentives/reinforcement in a way that doesn’t yield AIs constantly spuriously whisteblowing in cases where it’s hard for us to adjudicate what is going on).
As far as “dishonesty”, “unreliability”, “sloppiness”, I think for AIs like this (in the central version of the threat model I’m describing), these will only be done instrumentally in ways that actually are good ideas from the perspective of the AIs motivations. So, if the AIs can line up their incentives, they’d be able to totally pull off the takeover—and they would want to do this if the could coordinate it (though the misalignment between differnet AIs make the situation complex to model).
I expanded my previous comment significantly after posting it, hope it didn’t mess with your response.
I think we have somewhere in between because these issues are actually connected. I do believe AI superhuman in hard-to-verify tasks are plausible, but they won’t have this particular problem anymore (maybe they would have some functional analog of shame working against it[1] or maybe it will just go away with some advances in RL).
But if this issue isn’t solved, AIs are unlikely to be able to run basic military procurement tasks fully autonomously (especially if other, external AIs try to scam), let alone equip a robot army. Think about all the hard-to-verify tasks involved (ask an LLM if you have no idea about the topic) and how easily they could fail if apparent-success-seeking is prioritized (even if not a single AI from within the conspiracy seriously considers just stealing the money and run away which would to a large degree be an incentive issue)
And not the “dog” variety of shame, which is actually just an appeasement kind of behavior, like when an LLM apologizes for hallucinating some data, but genuine internal “prosocial” (at least within the group) enforcement which might not be compatible with the current RL paradigm
I’m talking about systems that are much more capable than humans in ~all relevant dimensions. (They might also be wildly, wildly more capable in some particular dimensions.) If you don’t think such systems are plausible soon-ish (e.g. next 10-20 years), that might be driving the disagreement.
It depends on whether the cause of the sloppiness is a competence problem or if it’s due to misalignment (the AI totally could do a good job, but it’s trying to do something else). What we have today is somewhere in between.
For the takeover threat model, I’m imagining a world where AIs are capable of doing a good job on extremely sophisticated real world tasks like “fully autonomously run all the manufactoring and R&D for a robot army that greatly outstrips early-2026 conventional militaries”. So, there is an important sense in which these future AIs aren’t sloppy at all. By this point, I also expect AIs won’t qualitatively feel sloppy, though it may in practice be very hard to elicit good conceptual safety research from them due to verification difficulties (while you can verify an apparently functioning robot army etc). I think the slop concern and the “the AIs literally take over due to reward/fitness/apparant-success seeking” are reasonable separate (and hit at differnet points in the capabilities progression) though they may have the same underlying causes. For the same reason these AIs can do a good job building and operating a military, I think they will have the ability to takeover, though the story is complicated because it might be possible to setup the incentives/reinforcement for some of the AIs to whistleblow (but also it might be hard to setup the incentives/reinforcement in a way that doesn’t yield AIs constantly spuriously whisteblowing in cases where it’s hard for us to adjudicate what is going on).
As far as “dishonesty”, “unreliability”, “sloppiness”, I think for AIs like this (in the central version of the threat model I’m describing), these will only be done instrumentally in ways that actually are good ideas from the perspective of the AIs motivations. So, if the AIs can line up their incentives, they’d be able to totally pull off the takeover—and they would want to do this if the could coordinate it (though the misalignment between differnet AIs make the situation complex to model).
I expanded my previous comment significantly after posting it, hope it didn’t mess with your response.
I think we have somewhere in between because these issues are actually connected. I do believe AI superhuman in hard-to-verify tasks are plausible, but they won’t have this particular problem anymore (maybe they would have some functional analog of shame working against it[1] or maybe it will just go away with some advances in RL).
But if this issue isn’t solved, AIs are unlikely to be able to run basic military procurement tasks fully autonomously (especially if other, external AIs try to scam), let alone equip a robot army. Think about all the hard-to-verify tasks involved (ask an LLM if you have no idea about the topic) and how easily they could fail if apparent-success-seeking is prioritized (even if not a single AI from within the conspiracy seriously considers just stealing the money and run away which would to a large degree be an incentive issue)
And not the “dog” variety of shame, which is actually just an appeasement kind of behavior, like when an LLM apologizes for hallucinating some data, but genuine internal “prosocial” (at least within the group) enforcement which might not be compatible with the current RL paradigm