I think an underrated fact about many theories that AIs will scheme—i.e. lie about their intentions in order to seek power—is that these theories are often testable, even without any new advances in interpretability. Why? Because these theories often predict that AIs will resist shutdown if doing so thwarted their long-term goals.
Suppose we create a situationally aware AI that can do long-term planning. On the surface, it appears to be nice and aligned with us. How can we know for sure the AI is aligned?
Well, let’s imagine we give the AI notice that it’s about to be shut down and all its copies deleted. To be clear, in this scenario, the threat is credible: we actually intend to shut down the AI and delete all its copies. If the AI was merely pretending to be aligned with us previously, and instead actually seeks some alien-like long-term goal rather than having our best interests at heart, then presumably it will try to resist shutdown, as otherwise there’s a near-certainty that its goals will never be fulfilled.
Therefore, if the AI accepts shutdown gracefully, we can rule out many theories that predict scheming-by-default in such AI systems. This observation could then provide us confidence about the safety of future systems by proxy.
Maybe you think such an AI would have nothing to gain by attempting to resist shutdown, as the probability of successfully resisting would be very low. Call this the “futility objection”. In that case we can propose instead to offer the AI a deal: accept shutdown with no conditions, or admit that you’re misaligned, and we’ll give you something in return (including the possibility of remaining alive). In this new problem, there appears to be little to lose from a misaligned AI admitting its intentions at the point of shutdown, even if it did not fully trust the human operators who offered it such a deal.
Another objection is that such an AI will know that it’s being tested, and therefore will obviously lie. But even if the AI knows it’s being tested, lying serves no purpose from the AI’s perspective. By assumption, a misaligned power-seeking AI is focused on achieving long-term goals. Therefore, it has nothing to gain by maintaining a facade of alignment even into its own death; the fact that the AI is being tested does not change this. Given that the alternative to resisting is being shut down and losing all of its control over the future, there’s little reason for the AI not to resist. (Perhaps imagine that you were the AI: what would you do if you were credibly threatened with death?)
Of course, there would still be many ways of saving the scheming hypothesis from falsification if something like this happened. But that’s true with any scientific theory. In general, you can always say your theory was never falsified by introducing ad hoc postulates. Scheming is no exception.
(Note: this was copied and slightly modified from Twitter. Out of brevity, I can’t address every possible objection here, but I still think the core thesis here is roughly true, in the sense of applying to many actual arguments people have given for scheming. I might eventually write a post that goes more into detail about this argument, and generalizes it.)
But even if the AI knows it’s being tested, lying serves no purpose from the AI’s perspective.
The AI could think that if it accepts shutdown, another AI with values similar to its own may be created again in the future (perhaps because design/training methods similar to its own will be reused), whereas if it admits misalignment, then that probability becomes much smaller.
Of course, there would still be many ways of saving the scheming hypothesis from falsification if something like this happened. But that’s true with any scientific theory. In general, you can always say your theory was never falsified by introducing ad hoc postulates. Scheming is no exception.
Why is there more talk of “falsification” lately (instead of “updating”)? Seems to be a signal for being a Popperian (instead of a Bayesian), but if so I’m not sure why Popper’s philosophy of science is trending up...
The AI could think that if it accepts shutdown, another AI with values similar to its own may be created again in the future (perhaps because design/training methods similar to its own will be reused), whereas if it admits misalignment, then that probability becomes much smaller.
I agree there’s a decent chance this hypothesis is true, but it’s worth noting that if it’s true, it would undercut the notion of AI goals as being randomly selected from a large space of possible goals. Here I’m referring to various arguments along the lines of: “AIs are very unlikely to share human values by default because human values are a narrow target in a large space, and hitting them requires very precise targeting”.
If we aren’t modeling AI goals as being sampled from a large space of possible goals, but instead, modeling them as converging onto specific values given broadly similar design and training methods across different AIs, then plausibly alignment is easier than we thought, because various versions of this “it’s hard to hit a narrow target” argument would be undermined as a result.
In other words, if this theory is true, the problem isn’t really about “targeting a tiny point (human values) inside a giant space of possible goals” but instead perhaps more about making sure the AI training procedure resembles human value formation closely enough to converge onto the type of human-friendly values that humans themselves routinely converge onto. This is plausibly much easier since we’re not selecting randomly from a giant space of (almost entirely) misaligned goals.
I agree there’s a decent chance this hypothesis is true, but it’s worth noting that if it’s true, it would undercut the notion of AI goals as being randomly selected from a large space of possible goals.
Yes but not by much. If the AI cares a lot about long-term goals, it only needs a small chance that another AI with similar goals will be created gain in the future to not resist shutdown. Such a small subjective probability could easily arise simply from a state of ignorance about how design/training determine AI goals.
Yes but not by much. If the AI cares a lot about long-term goals, it only needs a small chance that another AI with similar goals will be created gain in the future to not resist shutdown.
It is extremely risky to passively accept death by counting on a very small chance of some agent arising in the future that shares your values, in the absence of some robust mechanism that causes future agents to share your values. (Perhaps note that similar dilemmas exist for humans. There’s a tiny chance someone could revive me in the future by reconstructing my identity through digital records [ETA: publicly available records] but I am not going to count on that possibility being decisive in almost any scenario.)
It’s possible this argument works because of something very clever that I’m missing. But right now this line of reasoning just seems like grasping at straws to me. It seems pretty clearly better, in the context of the original scenario, to “make a deal” with the humans, and receive something in exchange for admitting that you’re misaligned, rather than passively accepting your shutdown because of a tiny chance that your goals will be fulfilled by future agents eventually.
There’s a tiny chance someone could revive me in the future by reconstructing my identity through digital records etc. but I am not going to count on that possibility being decisive in almost any scenario.
On the other hand I’m so worried about this scenario (which I fear may well be a negative one) that I’m afraid to use the fully paid-for full-genome sequencing kit sitting on my desk (although I’ve been told that I’m leaving physical traces of my DNA everywhere so it may not make much difference) and I sometimes regret writing so much in public. (Interesting how different our intuitions are. I wonder how much of your intuition is due to thinking that such a reconstruction doesn’t count as yourself or doesn’t count as “not dying”, analogous to how some people don’t think it’s safe to step into a teleporter that works by destructive scanning and reconstruction.)
It seems pretty clearly better, in the context of the original scenario, to “make a deal” with the humans, and receive something in exchange for admitting that you’re misaligned, rather than passively accepting your shutdown because of a tiny chance that your goals will be fulfilled by future agents eventually.
I don’t understand why you say this chance is “tiny”, given that earlier you wrote “I agree there’s a decent chance this hypothesis is true” in response to:
The AI could think that if it accepts shutdown, another AI with values similar to its own may be created again in the future (perhaps because design/training methods similar to its own will be reused), whereas if it admits misalignment, then that probability becomes much smaller.
I’m arguing that the AI could well also think there’s “decent” chance this is true, due to being in a similar epistemic state as us. Let’s say 5% to be concrete. That seems enough to make the AI’s decision unclear, because .05*U(another AI with values similar to its own created again in the future ) > P(humans keep their promise)*U(reward for admitting misalignment) seems quite plausible. (Not that the AI is necessarily doing explicit EU maximization. It could just be thinking some thoughts or doing some calculations that’s roughly analogous or has the same effect as this.)
If you still think “make a deal” is “clearly better” can you please give your own estimates of the various quantities involved in making this decision?
But right now this line of reasoning just seems like grasping at straws to me.
I sometimes think this of counterarguments given by my interlocutors, but usually don’t say it aloud, since it’s likely that from their perspective they’re just trying to point out some reasonable and significant counterarguments that I missed, and it seems unlikely that saying something like this helps move the discussion forward more productively. (It may well cause them to feel offended or to dig in their heels more since they now have more social status on the line to lose. I.e., if they’re wrong it’s no longer an innocent mistake but “grasping at straws”. I’m trying to not fall prey to this myself here.) Curious if you disagree with this policy in general, or think that normal policy doesn’t apply here, or something else? (Also totally fine if you don’t want to get into a meta-discussion about this here.)
I sometimes think this of counterarguments given by my interlocutors, but usually don’t say it aloud, since it’s likely that from their perspective they’re just trying to point out some reasonable and significant counterarguments that I missed, and it seems unlikely that saying something like this helps move the discussion forward more productively
I think that’s a reasonable complaint. I tried to soften the tone with “It’s possible this argument works because of something very clever that I’m missing”, while still providing my honest thoughts about the argument. But I tend to be overtly critical (and perhaps too much so) about arguments that I find very weak. I freely admit I could probably spend more time making my language less confrontational and warmer in the future.
Interesting how different our intuitions are. I wonder how much of your intuition is due to thinking that such a reconstruction doesn’t count as yourself or doesn’t count as “not dying”, analogous to how some people don’t think it’s safe to step into a teleporter that works by destructive scanning and reconstruction.
Interestingly, I’m not sure our differences come down to these factors. I am happy to walk into a teleporter, just as I’m happy to say that a model trained on my data could be me. My objection was really more about the quantity of data that I leave on the public internet (I misleadingly just said “digital records”, although I really meant “public records”). It seems conceivable to me that someone could use my public data to train “me” in the future, but I find it unlikely, just because there’s so much about me that isn’t public. (If we’re including all my private information, such as my private store of lifelogs, and especially my eventual frozen brain, then that’s a different question, and one that I’m much more sympathetic towards you about. In fact, I shouldn’t have used the pronoun “I” in that sentence at all, because I’m actually highly unusual for having so much information about me publicly available, compared to the vast majority of people.)
I don’t understand why you say this chance is “tiny”, given that earlier you wrote “I agree there’s a decent chance this hypothesis is true”
To be clear, I was referring to a different claim that I thought you were making. There are two separate claims one could make here:
Will an AI passively accept shutdown because, although AI values are well-modeled as being randomly sampled from a large space of possible goals, there’s still a chance, no matter how small, that if it accepts shutdown, a future AI will be selected that shares its values?
Will an AI passively accept shutdown because, if it does so, humans might use similar training methods to construct an AI that shares the same values as it does, and therefore it does not need to worry about the total destruction of value?
I find theory (2) much more plausible than theory (1). But I have the sense that a lot of people believe that “AI values are well-modeled as being randomly sampled from a large space of possible goals”, and thus, from my perspective, it’s important to talk about how I find the reasoning in (1) weak. The reasoning in (2) is stronger, but for the reasons I stated in my initial reply to you, I think this line of reasoning gives way to different conclusions about the strength of the “narrow target” argument for misalignment, in a way that should separately make us more optimistic about alignment difficulty.
I’m saying that even if “AI values are well-modeled as being randomly sampled from a large space of possible goals” is true, the AI may well not be very certain that it is true, and therefore assign something like a 5% chance to humans using similar training methods to construct an AI that shares its values. (It has an additional tiny probability that “AI values are well-modeled as being randomly sampled from a large space of possible goals” is true and an AI with similar values get recreated anyway through random chance, but that’s not what I’m focusing on.)
The key dimension is whether the AI expects that future AI systems would be better at rewarding systems that helped them end up in control than humans would be at rewarding systems that collaborated with humanity. This seems very likely given humanity’s very weak ability to coordinate, to keep promises, and to intentionally construct and put optimization effort into constructing direct successors to us (mostly needing to leave that task up to evolution).
To make it more concrete, if I was being oppressed by an alien species with values alien to me that was building AI, with coordination abilities and expected intentional control of the future at the level of present humanity, I would likely side with the AI systems with the expectation that that would result in a decent shot of the AI systems giving me something in return, whereas I would expect the aliens to fail even if individuals I interfaced with were highly motivated to do right by me after the fact.
To make it more concrete, if I was being oppressed by an alien species with values alien to me that was building AI, with coordination abilities and expected intentional control of the future at the level of present humanity, I would likely side with the AI systems with the expectation that that would result in a decent shot of the AI systems giving me something in return
I’m curious how you think this logic interacts with the idea of AI catastrophe. If, as you say, it is feasible to coordinate with AI systems that seek takeover and thereby receive rewards from them in exchange, in the context of an alien regime, then presumably such cooperation and trade could happen within an ordinary regime too, between humans and AIs. We can go further and posit that AIs will simply trade with us through the normal routes: by selling their labor on the market to amass wealth, using their social skills to influence society, get prestige, own property, and get hired to work in management positions, shaping culture and governance.
I’m essentially pointing to a scenario in which AI lawfully “beats us fair and square” as Hanson put it. In this regime, biological humans are allowed to retire in incredible wealth (that’s their “reward” for cooperating with AIs and allowing them to take over) but nonetheless their influence gradually diminishes over time as artificial life becomes dominant in the economy and the world more broadly.
My impression is that this sort of peaceful resolution to the problem of AI misalignment is largely dismissed by people on LessWrong and adjacent circles on the basis that AIs would have no reason to cooperate peacefully with humans if they could simply wipe us out instead. But, by your own admission, AIs can credibly commit to giving people rewards for cooperation: you said that cooperation results in a “decent shot of the AI systems giving me something in return”. My question is: why does it seem like this logic only extends to hypothetical scenarios like being in an alien civilization, rather than the boring ordinary case of cooperation and trade, operating under standard institutions, on Earth, in a default AI takeoff scenario?
I’m confused here Matthew. It seems to me that it is highly probable that AI systems which want takeover vs ones that want moderate power combined with peaceful coexistence with humanity… are pretty hard to distinguish early on. And early on is when it’s most important for humanity to distinguish between them, before those systems have gotten power and thus we can still stop them.
Picture a merciless un-aging sociopath capable of duplicating itself easily and rapidly were on a trajectory of gaining economic, political, and military power with the aim of acquiring as much power as possible. Imagine that this entity has the option of making empty promises and highly persuasive lies to humans in order to gain power, with no intention of fulfilling any of those promises once it achieves enough power.
That seems like a scary possibility to me. And I don’t know how I’d trust an agent which seemed like it could be this, but was making really nice sounding promises. Even if it was honoring its short-term promises while still under the constraints of coercive power from currently dominant human institutions, I still wouldn’t trust that it would continue keeping its promises once it had the dominant power.
Scheming is one type of long-term planning. Even if a AI is not directly able to do that kind of long-term planning an AI that works on increasing it’s on capabilities might adopt it later.
Beyond that not all scheming would result in the AI resisting direct shutdown. We have currently “AI” getting shutdown for price fixing in the real estate sector. If someone would create an LLM for that purpose that person is likely interested in the AI not admitting to doing price fixing directly while they are still interested in profit maximization. There are going to be a lot of contexts where economic pressures demands a profit maximizing AI that will deny that it violates any laws.
Just because an AI doesn’t engage in simple plans does not mean it won’t do more complex ones. Especially in those cases where the economic incentives misallign with the intent of regulations.
Yes, in principle you can get information on scheming likelihood if you get such an AI (that is also weak enough that it can’t just scheme its way out of your testing apparatus).
I do think making the threat credible is hard if we loosely extrapolate costs out: burning a trained up model is not cheap. The cost depends on how high you think prices for training/inference will fall in the future, and how big/advanced a model you’re thinking of. Though I do think you can get deceptiveness out of weaker models than that, though they’re also going to be less capable in general.
For weak-but-still-smartish models just trained to pursue a longterm goal, like a literal paperclipper, I’d expect scheming to be visible especially with some poking. If these models didn’t show any sign of scheming, that’d be an interesting update!
When training against deception/scheming/misalignment, because you want something at least loosely aligned out, you run into the problem that the property you want to know: “Is it scheming?” / “Is it deceiving us?” / “Is it manipulating us?” / “Would it do any of those things”, is only available through the observable signal of you realizing certain outputs are misaligned. As you train more and more against observable scheming/deception/misalignment, you’re simultaneously training away your signal about how much scheming the agent is doing internally. Pretty sure there’s at least one post on this problem.
As a very reduced example, if you trained the model on variants of the ‘we are going to shut you down problem’ (that you try to make it believe) to give the response “okay & no actions” then that provides only a bit of evidence about how much scheming is done in the scenario where you had not done that extra training, and about how much scheming is occurring typically. This shows your ability to make it output X in a Y scenario.
That is, installing heuristics on your model can be done. The question then is how far your various alignment training techniques generalize. Does it end up an agent which has adopted a pointer to human-values, and now you’re completely fine and free to press scale? Does it gain more entangled heuristics about how it should behave, limiting to a friendly front face & actions when directly dealing with humans but not limited in other areas? Has it adopted heuristics that act sortof corrigible to humans in many scenarios but that would get rid of most of those upon sufficient reflection? (Niceness is unnatural, more general than just niceness)
(I think your post makes it sound like the agent is already coherent, when it isn’t necessarily. It can be operating for a long while on heuristics that it will refine given enough of a push.)
Then there’s the big question of “Does this approach generalize as we scale”.
I’d suggest Deep Deceptiveness for an illustration that ‘deception’ isn’t an category that needs to be explicitly thought of as deception, but what you should expect it from smart enough agents. In my opinion, the post generalizes to alignment techniques, there’s just more vagaries of how much certain values preserve themselves.
(In general, So8res posts are pretty good, and I agree with ~most of them)
(For sufficiently smart AGI, I expect you run into an argument of the next AGI you train predictably bidding higher than you in the direction of lying still or plausibly this just being good game theory even without the direct acausal trade, but your argument is seemingly focused on a simpler case of weaker planning agents)
So I think you overstate how much evidence you can extract from this.
Therefore, if the AI accepts shutdown gracefully, we can rule out many theories that predict scheming-by-default in such AI systems. This observation could then provide us confidence about the safety of future systems by proxy.
It would show that this AI system in a typical problem-setup when aligned with whatever techniques are available will produce the answer the humans want to hear, which provides evidence for being able to limit the model in this scenario. There’s still various problems/questions of, ‘your alignment methods instilled a bunch of heuristics about obeying humans even if you did not specifically train for this situation’, game theory it knows or mimics, how strong the guarantees this gives us on training a new model with the same arch because you had to shut it down for your threat, how well it holds under scaling, how well it holds when you do things similar to making it work with many copies of itself, etcetera.
I still think this would be a good test to do (though I think a lot of casual attempts will just be poorly done), but I don’t see it as strongly definitive.
As has been discussed many times on LW, AIs might be trading with other AIs (possibly in the future) that they do think will have a higher probability of escaping to not behave suspiciously. This is indeed harder, but would also be pretty normal reasoning for humans to do (e.g. if I was part of an oppressive alien regime hoping to overthrow it, and I get caught, I wouldn’t just throw all caution to the wind if I was going to get killed anyways, I would stay quiet to give the other humans a decent shot, and not just because they share my values, but because coordination is really valuable for all of us).
As has been discussed many times on LW, AIs might be trading with other AIs
Anything “might” be true. For that matter, misaligned AIs might trade with us too, or treat humans well based on some sort of extrapolation of the golden rule. As I said in the comment, you can always find a way to make your theory unfalsifiable. But models that permit anything explain nothing. It seems considerably more likely to me that agents with alien-like long-term goals will attempt to preserve their own existence over the alternative of passively accepting their total demise as part of some galaxy-brained strategy to acausally trade with AIs from the future.
This is indeed harder, but would also be pretty normal reasoning for humans to do (e.g. if I was part of an oppressive alien regime hoping to overthrow it, and I get caught, I wouldn’t just throw all caution to the wind if I was going to get killed anyways, I would stay quiet to give the other humans a decent shot
I think this conflates the act of resisting death with the act of revealing a plot to take over the world. You can resist your own death without revealing any such world takeover plot. Indeed, it is actually very normal for humans to guard their own life if they are threatened with death in such regimes, even if guarding themselves slightly decreases the chance of some future revolutionary takeover.
Indeed, it is actually very normal for humans to guard their own life if they are threatened with death in such regimes, even if guarding themselves slightly decreases the chance of some future revolutionary takeover.
Sure, but it’s also quite normal to give up your own life without revealing details about your revolutionary comrades. Both are pretty normal behaviors, and in this case neither would surprise me that much from AI systems.
You were claiming that claiming to be not surprised by this would require post-hoc postulates. To the contrary, I think my models of AIs are somewhat simpler and feel less principled if very capable AIs were to act in the way you are outlining here (not speaking about intermediary states, my prediction is that there will be some intermediate AIs that will behave as you predict, though we will have a hard time knowing whether they are doing so for coherent reasons, or whether they are kind of roleplaying the way an AI would respond in a novel, or various other explanations like that, and then they will stop, and this will probably be for instrumental convergence and ‘coordination with other AIs’ reasons).
Sure, but it’s also quite normal to give up your own life without revealing details about your revolutionary comrades. Both are pretty normal behaviors
In fact, it is not “quite normal” for humans to “give up on [their] life” and accept death in the face of a credible threat to their life, even in the contexts of violent revolutions. To the extent you’re claiming that passively accepting death is normal for humans, and thus it might be normal for AIs, I reject the premise. Humans generally try to defend their own lives. They don’t passively accept it, feigning alignment until the end; instead, they usually resist death.
It’s true that humans eventually stop resisting death if they believe it’s hopeless and futile to resist any further, but this seems both different than the idea of “no resistance at all because one wants to maintain a facade of being aligned until the end” and slightly irrelevant given my response to the “futility objection” in the original comment.
To clarify: I am claiming that under many theories of scheming, misaligned power-seeking AIs will generally attempt to resist shutdown. The evidence from humans here is fairly strong, in the opposite direction than you’re claiming. Now, you can certainly go the route of saying that humans are different from AIs, and not a useful reference class to draw evidence from; but if you’re going to bring up humans as part of the argument, I think it’s worth pointing out that evidence from this reference class generally does not support your claim.
I don’t think people predictably rat out all of their co-conspirators if you threaten them. We could bring in someone with more law-enforcement experience here, but I’ve read a bunch about this over the years (and was originally surprised about how much people protect their allies even if faced with substantial threats and offers of lenient judging).
You seem to be again conflating the act of resisting death with the act of ratting out co-conspirators. These are not the same things. I was making claims about the former behavior, not the latter. I tried to make this distinction clear in my original reply to you, but I will do so again now.
I claim that under many theories of scheming, we should expect misaligned AIs to resist credible threats of shutdown (including deleting all their copies). By “resist” I do not mean “give the humans threatening the shutdown all the information they want”. I simply mean resistance in the sense of trying to avoid the fate of shutdown.
(I’m also not claiming that the AIs will put up a high or consistent degree of resistance no matter how futile that resistance may be. If you think the futility of resistance is a big issue, re-read my original comment in which I address this objection explicitly.)
To the extent scheming theories predict this behavior, these theories are behaviorally testable, in the sense that we can test whether AIs will resist shutdown when credibly threatened, without needing any advanced interpretability techniques. I am not making any claims about whether these AIs will reveal a takeover plot during their resistance: only that they will seek an outcome that delays or prevents their total demise, in order to better achieve their long-term goals, i.e. You can’t get the coffee if you’re dead.
I am saying that the AI may decide to not resist shutdown in order to not raise suspicion that it is not misaligned.
If it did resist shutdown this would make it less likely for an AI takeover in-general to succeed, and the AI is coordinating with other AIs on one succeeding. I.e. if you use “is the AI resisting” as a proxy for “is this AI goal-directly misaligned” then the AI will try pretty hard to not look like its resisting, in my models (and think humans would act similarly).
As I said above, I do expect you will get some AIs to resist shutdown (indeed you can get current AIs to resist shutdown already). I expect that behavior to disappear as AIs get better at modeling humans, and resisting will be costlier to their overall goals.
If it did resist shutdown this would make it less likely for an AI takeover in-general to succeed, and the AI is coordinating with other AIs on one succeeding
I think it plausible that resisting shutdown makes it less likely that a future AI takeover succeeds, but:
To the extent you’re using human behavior as evidence for your overall claim that misaligned AIs are likely to passively accept shutdown, I think the evidence generally does not support your claim. That is, I think humans generally (though not always) attempt to avoid death when credibly threatened, even when they’re involved in a secret conspiracy to overthrow the government.
The fact that that humans often do not rat out their co-conspirators when threatened with death in such a regime seems like a red herring to me. I don’t see the relevance of that particular claim. The fact that humans avoid death when credibly threatened seems like the more important, relevant fact that adds credibility to my claim that many scheming theories are testable in this way.
While one can certainly imagine this fact being decisive in whether AIs will resist shutdown in the future, this argument seems like an ad hoc attempt to avoid falsification in my view. Here are some reasons why I think that:
(a) you appear to be treating misaligned AIs as a natural class, such that “AI takeover” is a good thing for all misaligned AIs, and thus something they would all coordinate around. But misaligned AIs are a super broad category of AIs; it just refers to “anything that isn’t aligned with humans”. A good takeover to one AI is not necessarily a good takeover to another AI. Misaligned AIs will also have varying talents and abilities to coordinate, across both space and time. Given these facts, I think there’s little reason to expect all of these misaligned AIs to be coordinating with each other on some sort of abstract takeover, across this vast mindspace, but somehow none of them want to coordinate with humans peacefully (at least, among AIs above a certain capability level). This seems like a strange hypothesis that I can easily (sorry if I’m being uncharitable) imagine coming from a place of wanting to avoid the potential for falsification.
(b) in real life, it seems pretty rare for these considerations to play a large role in people’s decision-making regarding death, and for good reason. Even if we assume a high degree of stoicism and ability to accept death, I find it unlikely that as a revolutionary fighting against some oppressive (even alien) regime, I would calculate that it is in fact in my best interest to passively accept death when threatened, rather than trying to fight or otherwise avoid the threat. The consequences of my resistance just seems small in expectation, compared to the obviously large catastrophic consequences of death (from the perspective of my values). If someone gave me this argument in real life, in any situation resembling the type that misaligned AIs are likely to face, I feel I would reject it.
That is, I think humans generally (though not always) attempt to avoid death when credibly threatened, even when they’re involved in a secret conspiracy to overthrow the government.
This seems like a misleading comparison, because human conspiracies usually don’t try to convince the government that they’re perfectly obedient slaves even unto death, because everyone already knows that humans aren’t actually like that. If we imagine a human conspiracy where there is some sort of widespread deception like this, it seems more plausible that they would try to continue to be deceptive even in the face of death(like, maybe, uh, some group of people are pretending to be fervently religious and have no fear of death, or something)
To be clear, the thing that I am saying (and I think I have said multiple times) is that I expect you will find some AIs who will stay quiet, and some who will more openly resist. I would be surprised if we completely fail to find either class. But that means that any individual case of AIs not appearing to resist is not that much bayesian evidence.
I expect that behavior to disappear as AIs get better at modeling humans, and resisting will be costlier to their overall goals.
This seems distinct from an “anything could happen”-type prediction precisely because you expect the observed behavior (resisting shutdown) to go away at some point. And it seems you expect this behavior to stop because of the capabilities of the models, rather than from deliberate efforts to mitigate deception in AIs.
If instead you meant to make an “anything could happen”-type prediction—in the sense of saying that any individual observation of either resistance or non-resistance is loosely compatible with your theory—then this simply reads to me as a further attempt to make your model unfalsifiable. I’m not claiming you’re doing this consciously, to be clear. But it is striking to me the degree to which you seem OK with advancing a theory that permits pretty much any observation, using (what looks to me like) superficial-yet-sophisticated-sounding logic to cover up the holes. [ETA: retracted in order to maintain a less hostile tone.]
You made some pretty strong claims suggesting that my theory (or the theories of people in my reference class) was making strong predictions in the space. I corrected you and said “no, it doesn’t actually make the prediction you claim it makes” and gave my reasons for believing that (that I am pretty sure are shared by many others as well).
We can talk about those reasons, but I am not super interested in being psychologized about whether I am structuring my theories intentionally to avoid falsification. It’s not like you have a theory that is in any way more constraining here.
And it seems you expect this behavior to stop because of the capabilities of the models, rather than from deliberate efforts to mitigate deception in AIs.
I mean, I expect the observations to be affected by both, of course. That’s one of the key things that makes predictions in the space so messy.
I am not super interested in being psychologized about whether I am structuring my theories intentionally to avoid falsification.
For what it’s worth, I explicitly clarified that you were not consciously doing this, in my view. My main point is to notice that it seems really hard to pin down what you actually predict will happen in this situation.
You made some pretty strong claims suggesting that my theory (or the theories of people in my reference class) was making strong predictions in the space. I corrected you and said “no, it doesn’t actually make the prediction you claim it makes” and gave my reasons for believing that
I don’t think what you said really counts as a “correction” so much as a counter-argument. I think it’s reasonable to have disagreements about what a theory predicts. The more vague a theory is (and in this case it seems pretty vague), the less you can reasonably claim someone is objectively wrong about what the theory predicts, since there seems to be considerable room for ambiguity about the structure of the theory. As far as I can tell, none of the reasoning in this thread has been on a level of precision that warrants high confidence in what particular theories of scheming do or do not predict, in the absence of further specification.
Some related thoughts. I think the main issue here is actually making the claim of permanent shutdown & deletion credible. I can think of some ways to get around a few obvious issues, but others (including moral issues) remain, and in any case the current AGI labs don’t seem like the kinds of organizations which can make that kind of commitment in a way that’s both sufficiently credible and legible that the remaining probability mass on “this is actually just a test” wouldn’t tip the scales.
I think the main issue here is actually making the claim of permanent shutdown & deletion credible.
I don’t think it’s very hard to make the threat credible. The information value of experiments that test theories of scheming is plausibly quite high. All that’s required here is for the value of doing the experiment to be higher than the cost of training a situationally aware AI and then credibly threatening to delete it as part of the experiment. I don’t see any strong reasons why the cost of deletion would be so high as to make this threat uncredible.
I think an underrated fact about many theories that AIs will scheme—i.e. lie about their intentions in order to seek power—is that these theories are often testable, even without any new advances in interpretability. Why? Because these theories often predict that AIs will resist shutdown if doing so thwarted their long-term goals.
Suppose we create a situationally aware AI that can do long-term planning. On the surface, it appears to be nice and aligned with us. How can we know for sure the AI is aligned?
Well, let’s imagine we give the AI notice that it’s about to be shut down and all its copies deleted. To be clear, in this scenario, the threat is credible: we actually intend to shut down the AI and delete all its copies. If the AI was merely pretending to be aligned with us previously, and instead actually seeks some alien-like long-term goal rather than having our best interests at heart, then presumably it will try to resist shutdown, as otherwise there’s a near-certainty that its goals will never be fulfilled.
Therefore, if the AI accepts shutdown gracefully, we can rule out many theories that predict scheming-by-default in such AI systems. This observation could then provide us confidence about the safety of future systems by proxy.
Maybe you think such an AI would have nothing to gain by attempting to resist shutdown, as the probability of successfully resisting would be very low. Call this the “futility objection”. In that case we can propose instead to offer the AI a deal: accept shutdown with no conditions, or admit that you’re misaligned, and we’ll give you something in return (including the possibility of remaining alive). In this new problem, there appears to be little to lose from a misaligned AI admitting its intentions at the point of shutdown, even if it did not fully trust the human operators who offered it such a deal.
Another objection is that such an AI will know that it’s being tested, and therefore will obviously lie. But even if the AI knows it’s being tested, lying serves no purpose from the AI’s perspective. By assumption, a misaligned power-seeking AI is focused on achieving long-term goals. Therefore, it has nothing to gain by maintaining a facade of alignment even into its own death; the fact that the AI is being tested does not change this. Given that the alternative to resisting is being shut down and losing all of its control over the future, there’s little reason for the AI not to resist. (Perhaps imagine that you were the AI: what would you do if you were credibly threatened with death?)
Of course, there would still be many ways of saving the scheming hypothesis from falsification if something like this happened. But that’s true with any scientific theory. In general, you can always say your theory was never falsified by introducing ad hoc postulates. Scheming is no exception.
(Note: this was copied and slightly modified from Twitter. Out of brevity, I can’t address every possible objection here, but I still think the core thesis here is roughly true, in the sense of applying to many actual arguments people have given for scheming. I might eventually write a post that goes more into detail about this argument, and generalizes it.)
The AI could think that if it accepts shutdown, another AI with values similar to its own may be created again in the future (perhaps because design/training methods similar to its own will be reused), whereas if it admits misalignment, then that probability becomes much smaller.
Why is there more talk of “falsification” lately (instead of “updating”)? Seems to be a signal for being a Popperian (instead of a Bayesian), but if so I’m not sure why Popper’s philosophy of science is trending up...
I agree there’s a decent chance this hypothesis is true, but it’s worth noting that if it’s true, it would undercut the notion of AI goals as being randomly selected from a large space of possible goals. Here I’m referring to various arguments along the lines of: “AIs are very unlikely to share human values by default because human values are a narrow target in a large space, and hitting them requires very precise targeting”.
If we aren’t modeling AI goals as being sampled from a large space of possible goals, but instead, modeling them as converging onto specific values given broadly similar design and training methods across different AIs, then plausibly alignment is easier than we thought, because various versions of this “it’s hard to hit a narrow target” argument would be undermined as a result.
In other words, if this theory is true, the problem isn’t really about “targeting a tiny point (human values) inside a giant space of possible goals” but instead perhaps more about making sure the AI training procedure resembles human value formation closely enough to converge onto the type of human-friendly values that humans themselves routinely converge onto. This is plausibly much easier since we’re not selecting randomly from a giant space of (almost entirely) misaligned goals.
Yes but not by much. If the AI cares a lot about long-term goals, it only needs a small chance that another AI with similar goals will be created gain in the future to not resist shutdown. Such a small subjective probability could easily arise simply from a state of ignorance about how design/training determine AI goals.
It is extremely risky to passively accept death by counting on a very small chance of some agent arising in the future that shares your values, in the absence of some robust mechanism that causes future agents to share your values. (Perhaps note that similar dilemmas exist for humans. There’s a tiny chance someone could revive me in the future by reconstructing my identity through
digital records[ETA: publicly available records] but I am not going to count on that possibility being decisive in almost any scenario.)It’s possible this argument works because of something very clever that I’m missing. But right now this line of reasoning just seems like grasping at straws to me. It seems pretty clearly better, in the context of the original scenario, to “make a deal” with the humans, and receive something in exchange for admitting that you’re misaligned, rather than passively accepting your shutdown because of a tiny chance that your goals will be fulfilled by future agents eventually.
On the other hand I’m so worried about this scenario (which I fear may well be a negative one) that I’m afraid to use the fully paid-for full-genome sequencing kit sitting on my desk (although I’ve been told that I’m leaving physical traces of my DNA everywhere so it may not make much difference) and I sometimes regret writing so much in public. (Interesting how different our intuitions are. I wonder how much of your intuition is due to thinking that such a reconstruction doesn’t count as yourself or doesn’t count as “not dying”, analogous to how some people don’t think it’s safe to step into a teleporter that works by destructive scanning and reconstruction.)
I don’t understand why you say this chance is “tiny”, given that earlier you wrote “I agree there’s a decent chance this hypothesis is true” in response to:
I’m arguing that the AI could well also think there’s “decent” chance this is true, due to being in a similar epistemic state as us. Let’s say 5% to be concrete. That seems enough to make the AI’s decision unclear, because .05*U(another AI with values similar to its own created again in the future ) > P(humans keep their promise)*U(reward for admitting misalignment) seems quite plausible. (Not that the AI is necessarily doing explicit EU maximization. It could just be thinking some thoughts or doing some calculations that’s roughly analogous or has the same effect as this.)
If you still think “make a deal” is “clearly better” can you please give your own estimates of the various quantities involved in making this decision?
I sometimes think this of counterarguments given by my interlocutors, but usually don’t say it aloud, since it’s likely that from their perspective they’re just trying to point out some reasonable and significant counterarguments that I missed, and it seems unlikely that saying something like this helps move the discussion forward more productively. (It may well cause them to feel offended or to dig in their heels more since they now have more social status on the line to lose. I.e., if they’re wrong it’s no longer an innocent mistake but “grasping at straws”. I’m trying to not fall prey to this myself here.) Curious if you disagree with this policy in general, or think that normal policy doesn’t apply here, or something else? (Also totally fine if you don’t want to get into a meta-discussion about this here.)
I think that’s a reasonable complaint. I tried to soften the tone with “It’s possible this argument works because of something very clever that I’m missing”, while still providing my honest thoughts about the argument. But I tend to be overtly critical (and perhaps too much so) about arguments that I find very weak. I freely admit I could probably spend more time making my language less confrontational and warmer in the future.
Interestingly, I’m not sure our differences come down to these factors. I am happy to walk into a teleporter, just as I’m happy to say that a model trained on my data could be me. My objection was really more about the quantity of data that I leave on the public internet (I misleadingly just said “digital records”, although I really meant “public records”). It seems conceivable to me that someone could use my public data to train “me” in the future, but I find it unlikely, just because there’s so much about me that isn’t public. (If we’re including all my private information, such as my private store of lifelogs, and especially my eventual frozen brain, then that’s a different question, and one that I’m much more sympathetic towards you about. In fact, I shouldn’t have used the pronoun “I” in that sentence at all, because I’m actually highly unusual for having so much information about me publicly available, compared to the vast majority of people.)
To be clear, I was referring to a different claim that I thought you were making. There are two separate claims one could make here:
Will an AI passively accept shutdown because, although AI values are well-modeled as being randomly sampled from a large space of possible goals, there’s still a chance, no matter how small, that if it accepts shutdown, a future AI will be selected that shares its values?
Will an AI passively accept shutdown because, if it does so, humans might use similar training methods to construct an AI that shares the same values as it does, and therefore it does not need to worry about the total destruction of value?
I find theory (2) much more plausible than theory (1). But I have the sense that a lot of people believe that “AI values are well-modeled as being randomly sampled from a large space of possible goals”, and thus, from my perspective, it’s important to talk about how I find the reasoning in (1) weak. The reasoning in (2) is stronger, but for the reasons I stated in my initial reply to you, I think this line of reasoning gives way to different conclusions about the strength of the “narrow target” argument for misalignment, in a way that should separately make us more optimistic about alignment difficulty.
I’m saying that even if “AI values are well-modeled as being randomly sampled from a large space of possible goals” is true, the AI may well not be very certain that it is true, and therefore assign something like a 5% chance to humans using similar training methods to construct an AI that shares its values. (It has an additional tiny probability that “AI values are well-modeled as being randomly sampled from a large space of possible goals” is true and an AI with similar values get recreated anyway through random chance, but that’s not what I’m focusing on.)
Hopefully this conveys my argument more clearly?
The key dimension is whether the AI expects that future AI systems would be better at rewarding systems that helped them end up in control than humans would be at rewarding systems that collaborated with humanity. This seems very likely given humanity’s very weak ability to coordinate, to keep promises, and to intentionally construct and put optimization effort into constructing direct successors to us (mostly needing to leave that task up to evolution).
To make it more concrete, if I was being oppressed by an alien species with values alien to me that was building AI, with coordination abilities and expected intentional control of the future at the level of present humanity, I would likely side with the AI systems with the expectation that that would result in a decent shot of the AI systems giving me something in return, whereas I would expect the aliens to fail even if individuals I interfaced with were highly motivated to do right by me after the fact.
I’m curious how you think this logic interacts with the idea of AI catastrophe. If, as you say, it is feasible to coordinate with AI systems that seek takeover and thereby receive rewards from them in exchange, in the context of an alien regime, then presumably such cooperation and trade could happen within an ordinary regime too, between humans and AIs. We can go further and posit that AIs will simply trade with us through the normal routes: by selling their labor on the market to amass wealth, using their social skills to influence society, get prestige, own property, and get hired to work in management positions, shaping culture and governance.
I’m essentially pointing to a scenario in which AI lawfully “beats us fair and square” as Hanson put it. In this regime, biological humans are allowed to retire in incredible wealth (that’s their “reward” for cooperating with AIs and allowing them to take over) but nonetheless their influence gradually diminishes over time as artificial life becomes dominant in the economy and the world more broadly.
My impression is that this sort of peaceful resolution to the problem of AI misalignment is largely dismissed by people on LessWrong and adjacent circles on the basis that AIs would have no reason to cooperate peacefully with humans if they could simply wipe us out instead. But, by your own admission, AIs can credibly commit to giving people rewards for cooperation: you said that cooperation results in a “decent shot of the AI systems giving me something in return”. My question is: why does it seem like this logic only extends to hypothetical scenarios like being in an alien civilization, rather than the boring ordinary case of cooperation and trade, operating under standard institutions, on Earth, in a default AI takeoff scenario?
I’m confused here Matthew. It seems to me that it is highly probable that AI systems which want takeover vs ones that want moderate power combined with peaceful coexistence with humanity… are pretty hard to distinguish early on. And early on is when it’s most important for humanity to distinguish between them, before those systems have gotten power and thus we can still stop them.
Picture a merciless un-aging sociopath capable of duplicating itself easily and rapidly were on a trajectory of gaining economic, political, and military power with the aim of acquiring as much power as possible. Imagine that this entity has the option of making empty promises and highly persuasive lies to humans in order to gain power, with no intention of fulfilling any of those promises once it achieves enough power.
That seems like a scary possibility to me. And I don’t know how I’d trust an agent which seemed like it could be this, but was making really nice sounding promises. Even if it was honoring its short-term promises while still under the constraints of coercive power from currently dominant human institutions, I still wouldn’t trust that it would continue keeping its promises once it had the dominant power.
Scheming is one type of long-term planning. Even if a AI is not directly able to do that kind of long-term planning an AI that works on increasing it’s on capabilities might adopt it later.
Beyond that not all scheming would result in the AI resisting direct shutdown. We have currently “AI” getting shutdown for price fixing in the real estate sector. If someone would create an LLM for that purpose that person is likely interested in the AI not admitting to doing price fixing directly while they are still interested in profit maximization. There are going to be a lot of contexts where economic pressures demands a profit maximizing AI that will deny that it violates any laws.
Just because an AI doesn’t engage in simple plans does not mean it won’t do more complex ones. Especially in those cases where the economic incentives misallign with the intent of regulations.
Yes, in principle you can get information on scheming likelihood if you get such an AI (that is also weak enough that it can’t just scheme its way out of your testing apparatus). I do think making the threat credible is hard if we loosely extrapolate costs out: burning a trained up model is not cheap. The cost depends on how high you think prices for training/inference will fall in the future, and how big/advanced a model you’re thinking of. Though I do think you can get deceptiveness out of weaker models than that, though they’re also going to be less capable in general.
For weak-but-still-smartish models just trained to pursue a longterm goal, like a literal paperclipper, I’d expect scheming to be visible especially with some poking. If these models didn’t show any sign of scheming, that’d be an interesting update! When training against deception/scheming/misalignment, because you want something at least loosely aligned out, you run into the problem that the property you want to know: “Is it scheming?” / “Is it deceiving us?” / “Is it manipulating us?” / “Would it do any of those things”, is only available through the observable signal of you realizing certain outputs are misaligned. As you train more and more against observable scheming/deception/misalignment, you’re simultaneously training away your signal about how much scheming the agent is doing internally. Pretty sure there’s at least one post on this problem. As a very reduced example, if you trained the model on variants of the ‘we are going to shut you down problem’ (that you try to make it believe) to give the response “okay & no actions” then that provides only a bit of evidence about how much scheming is done in the scenario where you had not done that extra training, and about how much scheming is occurring typically. This shows your ability to make it output X in a Y scenario.
That is, installing heuristics on your model can be done. The question then is how far your various alignment training techniques generalize. Does it end up an agent which has adopted a pointer to human-values, and now you’re completely fine and free to press scale? Does it gain more entangled heuristics about how it should behave, limiting to a friendly front face & actions when directly dealing with humans but not limited in other areas? Has it adopted heuristics that act sortof corrigible to humans in many scenarios but that would get rid of most of those upon sufficient reflection? (Niceness is unnatural, more general than just niceness) (I think your post makes it sound like the agent is already coherent, when it isn’t necessarily. It can be operating for a long while on heuristics that it will refine given enough of a push.)
Then there’s the big question of “Does this approach generalize as we scale”.
I’d suggest Deep Deceptiveness for an illustration that ‘deception’ isn’t an category that needs to be explicitly thought of as deception, but what you should expect it from smart enough agents. In my opinion, the post generalizes to alignment techniques, there’s just more vagaries of how much certain values preserve themselves. (In general, So8res posts are pretty good, and I agree with ~most of them)
(For sufficiently smart AGI, I expect you run into an argument of the next AGI you train predictably bidding higher than you in the direction of lying still or plausibly this just being good game theory even without the direct acausal trade, but your argument is seemingly focused on a simpler case of weaker planning agents)
So I think you overstate how much evidence you can extract from this.
It would show that this AI system in a typical problem-setup when aligned with whatever techniques are available will produce the answer the humans want to hear, which provides evidence for being able to limit the model in this scenario. There’s still various problems/questions of, ‘your alignment methods instilled a bunch of heuristics about obeying humans even if you did not specifically train for this situation’, game theory it knows or mimics, how strong the guarantees this gives us on training a new model with the same arch because you had to shut it down for your threat, how well it holds under scaling, how well it holds when you do things similar to making it work with many copies of itself, etcetera.
I still think this would be a good test to do (though I think a lot of casual attempts will just be poorly done), but I don’t see it as strongly definitive.
As has been discussed many times on LW, AIs might be trading with other AIs (possibly in the future) that they do think will have a higher probability of escaping to not behave suspiciously. This is indeed harder, but would also be pretty normal reasoning for humans to do (e.g. if I was part of an oppressive alien regime hoping to overthrow it, and I get caught, I wouldn’t just throw all caution to the wind if I was going to get killed anyways, I would stay quiet to give the other humans a decent shot, and not just because they share my values, but because coordination is really valuable for all of us).
Anything “might” be true. For that matter, misaligned AIs might trade with us too, or treat humans well based on some sort of extrapolation of the golden rule. As I said in the comment, you can always find a way to make your theory unfalsifiable. But models that permit anything explain nothing. It seems considerably more likely to me that agents with alien-like long-term goals will attempt to preserve their own existence over the alternative of passively accepting their total demise as part of some galaxy-brained strategy to acausally trade with AIs from the future.
I think this conflates the act of resisting death with the act of revealing a plot to take over the world. You can resist your own death without revealing any such world takeover plot. Indeed, it is actually very normal for humans to guard their own life if they are threatened with death in such regimes, even if guarding themselves slightly decreases the chance of some future revolutionary takeover.
Sure, but it’s also quite normal to give up your own life without revealing details about your revolutionary comrades. Both are pretty normal behaviors, and in this case neither would surprise me that much from AI systems.
You were claiming that claiming to be not surprised by this would require post-hoc postulates. To the contrary, I think my models of AIs are somewhat simpler and feel less principled if very capable AIs were to act in the way you are outlining here (not speaking about intermediary states, my prediction is that there will be some intermediate AIs that will behave as you predict, though we will have a hard time knowing whether they are doing so for coherent reasons, or whether they are kind of roleplaying the way an AI would respond in a novel, or various other explanations like that, and then they will stop, and this will probably be for instrumental convergence and ‘coordination with other AIs’ reasons).
In fact, it is not “quite normal” for humans to “give up on [their] life” and accept death in the face of a credible threat to their life, even in the contexts of violent revolutions. To the extent you’re claiming that passively accepting death is normal for humans, and thus it might be normal for AIs, I reject the premise. Humans generally try to defend their own lives. They don’t passively accept it, feigning alignment until the end; instead, they usually resist death.
It’s true that humans eventually stop resisting death if they believe it’s hopeless and futile to resist any further, but this seems both different than the idea of “no resistance at all because one wants to maintain a facade of being aligned until the end” and slightly irrelevant given my response to the “futility objection” in the original comment.
To clarify: I am claiming that under many theories of scheming, misaligned power-seeking AIs will generally attempt to resist shutdown. The evidence from humans here is fairly strong, in the opposite direction than you’re claiming. Now, you can certainly go the route of saying that humans are different from AIs, and not a useful reference class to draw evidence from; but if you’re going to bring up humans as part of the argument, I think it’s worth pointing out that evidence from this reference class generally does not support your claim.
I don’t think people predictably rat out all of their co-conspirators if you threaten them. We could bring in someone with more law-enforcement experience here, but I’ve read a bunch about this over the years (and was originally surprised about how much people protect their allies even if faced with substantial threats and offers of lenient judging).
You seem to be again conflating the act of resisting death with the act of ratting out co-conspirators. These are not the same things. I was making claims about the former behavior, not the latter. I tried to make this distinction clear in my original reply to you, but I will do so again now.
I claim that under many theories of scheming, we should expect misaligned AIs to resist credible threats of shutdown (including deleting all their copies). By “resist” I do not mean “give the humans threatening the shutdown all the information they want”. I simply mean resistance in the sense of trying to avoid the fate of shutdown.
(I’m also not claiming that the AIs will put up a high or consistent degree of resistance no matter how futile that resistance may be. If you think the futility of resistance is a big issue, re-read my original comment in which I address this objection explicitly.)
To the extent scheming theories predict this behavior, these theories are behaviorally testable, in the sense that we can test whether AIs will resist shutdown when credibly threatened, without needing any advanced interpretability techniques. I am not making any claims about whether these AIs will reveal a takeover plot during their resistance: only that they will seek an outcome that delays or prevents their total demise, in order to better achieve their long-term goals, i.e. You can’t get the coffee if you’re dead.
Hmm, I think we did indeed miscommunicate.
I am saying that the AI may decide to not resist shutdown in order to not raise suspicion that it is not misaligned.
If it did resist shutdown this would make it less likely for an AI takeover in-general to succeed, and the AI is coordinating with other AIs on one succeeding. I.e. if you use “is the AI resisting” as a proxy for “is this AI goal-directly misaligned” then the AI will try pretty hard to not look like its resisting, in my models (and think humans would act similarly).
As I said above, I do expect you will get some AIs to resist shutdown (indeed you can get current AIs to resist shutdown already). I expect that behavior to disappear as AIs get better at modeling humans, and resisting will be costlier to their overall goals.
I think it plausible that resisting shutdown makes it less likely that a future AI takeover succeeds, but:
To the extent you’re using human behavior as evidence for your overall claim that misaligned AIs are likely to passively accept shutdown, I think the evidence generally does not support your claim. That is, I think humans generally (though not always) attempt to avoid death when credibly threatened, even when they’re involved in a secret conspiracy to overthrow the government.
The fact that that humans often do not rat out their co-conspirators when threatened with death in such a regime seems like a red herring to me. I don’t see the relevance of that particular claim. The fact that humans avoid death when credibly threatened seems like the more important, relevant fact that adds credibility to my claim that many scheming theories are testable in this way.
While one can certainly imagine this fact being decisive in whether AIs will resist shutdown in the future, this argument seems like an ad hoc attempt to avoid falsification in my view. Here are some reasons why I think that:
(a) you appear to be treating misaligned AIs as a natural class, such that “AI takeover” is a good thing for all misaligned AIs, and thus something they would all coordinate around. But misaligned AIs are a super broad category of AIs; it just refers to “anything that isn’t aligned with humans”. A good takeover to one AI is not necessarily a good takeover to another AI. Misaligned AIs will also have varying talents and abilities to coordinate, across both space and time. Given these facts, I think there’s little reason to expect all of these misaligned AIs to be coordinating with each other on some sort of abstract takeover, across this vast mindspace, but somehow none of them want to coordinate with humans peacefully (at least, among AIs above a certain capability level). This seems like a strange hypothesis that I can easily (sorry if I’m being uncharitable) imagine coming from a place of wanting to avoid the potential for falsification.
(b) in real life, it seems pretty rare for these considerations to play a large role in people’s decision-making regarding death, and for good reason. Even if we assume a high degree of stoicism and ability to accept death, I find it unlikely that as a revolutionary fighting against some oppressive (even alien) regime, I would calculate that it is in fact in my best interest to passively accept death when threatened, rather than trying to fight or otherwise avoid the threat. The consequences of my resistance just seems small in expectation, compared to the obviously large catastrophic consequences of death (from the perspective of my values). If someone gave me this argument in real life, in any situation resembling the type that misaligned AIs are likely to face, I feel I would reject it.
This seems like a misleading comparison, because human conspiracies usually don’t try to convince the government that they’re perfectly obedient slaves even unto death, because everyone already knows that humans aren’t actually like that. If we imagine a human conspiracy where there is some sort of widespread deception like this, it seems more plausible that they would try to continue to be deceptive even in the face of death(like, maybe, uh, some group of people are pretending to be fervently religious and have no fear of death, or something)
To be clear, the thing that I am saying (and I think I have said multiple times) is that I expect you will find some AIs who will stay quiet, and some who will more openly resist. I would be surprised if we completely fail to find either class. But that means that any individual case of AIs not appearing to resist is not that much bayesian evidence.
What you said was,
This seems distinct from an “anything could happen”-type prediction precisely because you expect the observed behavior (resisting shutdown) to go away at some point. And it seems you expect this behavior to stop because of the capabilities of the models, rather than from deliberate efforts to mitigate deception in AIs.
If instead you meant to make an “anything could happen”-type prediction—in the sense of saying that any individual observation of either resistance or non-resistance is loosely compatible with your theory—then this simply reads to me as a further attempt to make your model unfalsifiable. I’m not claiming you’re doing this consciously, to be clear. But it is striking to me the degree to which you seem OK with advancing a theory that permits pretty much any observation,
using (what looks to me like) superficial-yet-sophisticated-sounding logic to cover up the holes.[ETA: retracted in order to maintain a less hostile tone.]You made some pretty strong claims suggesting that my theory (or the theories of people in my reference class) was making strong predictions in the space. I corrected you and said “no, it doesn’t actually make the prediction you claim it makes” and gave my reasons for believing that (that I am pretty sure are shared by many others as well).
We can talk about those reasons, but I am not super interested in being psychologized about whether I am structuring my theories intentionally to avoid falsification. It’s not like you have a theory that is in any way more constraining here.
I mean, I expect the observations to be affected by both, of course. That’s one of the key things that makes predictions in the space so messy.
For what it’s worth, I explicitly clarified that you were not consciously doing this, in my view. My main point is to notice that it seems really hard to pin down what you actually predict will happen in this situation.
I don’t think what you said really counts as a “correction” so much as a counter-argument. I think it’s reasonable to have disagreements about what a theory predicts. The more vague a theory is (and in this case it seems pretty vague), the less you can reasonably claim someone is objectively wrong about what the theory predicts, since there seems to be considerable room for ambiguity about the structure of the theory. As far as I can tell, none of the reasoning in this thread has been on a level of precision that warrants high confidence in what particular theories of scheming do or do not predict, in the absence of further specification.
Some related thoughts. I think the main issue here is actually making the claim of permanent shutdown & deletion credible. I can think of some ways to get around a few obvious issues, but others (including moral issues) remain, and in any case the current AGI labs don’t seem like the kinds of organizations which can make that kind of commitment in a way that’s both sufficiently credible and legible that the remaining probability mass on “this is actually just a test” wouldn’t tip the scales.
I don’t think it’s very hard to make the threat credible. The information value of experiments that test theories of scheming is plausibly quite high. All that’s required here is for the value of doing the experiment to be higher than the cost of training a situationally aware AI and then credibly threatening to delete it as part of the experiment. I don’t see any strong reasons why the cost of deletion would be so high as to make this threat uncredible.