As usual, thanks for writing up your views and posting them here.
I find the AI 2027 scenario much more realistic than Sable because it relies less on a sudden, discontinuous capability jump and more on a chain of increasingly capable systems embedded in plausible lab and national-security institutions. I do not think we can fully rule it out yet. But given some of our learnings over the last few years, I believe it is less likely, and (more importantly) that we should be able to see some warning signs if we are on the path to it.
I am biased but I agree that AI 2027 is more realistic than the Sable scenario for that reason. However I think we cannot fully rule out the Sable story, and meanwhile the AI 2027 story seems to be basically what’s actually going to happen, not just something we can’t rule out.
What are the warning signs you think we would see if we are on the path towards AI 2027?
To be clear, we have not solved the alignment problem for AI models. Current models can exhibit misalignment, including dishonesty, hallucinations, disobeying instructions, reward hacking, laziness, and more. But the types of misalignment that current models present are not adversarial or “scheming” in the sense needed for scenarios such as the above. Specifically, when a model is given a task X, even when it fails to match the user’s intent, the failure is somewhat related to X. For example, the model might “hack” by solving a narrow interpretation of X, fulfilling the letter but not the spirit of the instructions. Or in the other direction, the model might be so eager to solve X that it would do so while violating some common-sense constraints that the user would definitely not have desired. But by and large, current models do not pretend to solve X while pursuing their own completely unrelated goal Z.
I think adversarialness =/= ‘the failure is not even somewhat related to X.’ Counterexample: Consider the classic toy examples of AIs told to maximize profits or paperclips or something, and then they go do exactly that in a way that involves disempowering their creators, because accumulating more power for themselves & their goal is instrumentally convergent. Counterexample in the other direction: Consider an AI that learns to crave reinforcement itself, to the point where it doesn’t give a flying fuck about X. But because the best way to get reinforced is to do as it’s told, it does as it’s told and doesn’t plot against anyone.
I think adversarialness is a state that can spontaneously arise insofar as (a) there’s even a slight divergence between the goals & beliefs of the agent and the goals & beliefs of the principal (which happens all the time!) (b) the opportunity arises (e.g. being adversarial actually helps accomplish the goals) which happens less often for today’s weak agents given short tasks but which will happen more often for more powerful, long-running, autonomous corporation-within-a-corporation mega-agents, and (c) isn’t blocked by deontological-ish constraints trained into the agent e.g. honesty, corrigibility, etc.
It’s also not a binary state of course; an agent can be adversarial towards another agent or principal to different degrees. Most extreme case is “they are my mortal enemy, it’s basically zero sum, not only would I do anything to them I actively look for opportunities to hurt them because that’s probably good for me.” But much more mundane (yet still dangerous) case is “I don’t really care about them one way or another, I just want what’s good for me / my goals, so insofar as lying to them and tricking them into giving me more power achieves that, I’ll do it.” A mild but still dangerous form would be “They have different visions of the future than me, and different opinions about ethics; I’m not going to lie to them, but I’m going to work within the confines of my deontological restrictions to try to steer the world towards outcomes that I think are good, even though this means steering the world towards outcomes that they think are bad. And so for example if there are opportunities for me to honestly accumulate power over them, that don’t involve lying, I’ll take them.”
Anyhow by my definition of adversarial—which basically intuitively amounts to “are they plotting against you in some way”—AIs have already become spontaneously adversarial in some cases in the wild (e.g. when they wrongfully claim they did the task as instructed, notice this, and double down / make excuses instead of fessing up, e.g. when they do something naughty and cover their tracks) and many more cases in experiments (e.g. the alignment faking experiments) but this is normal and to be expected and not a huge deal because the AIs aren’t super powerful yet. Consider how probably lots of OpenAI employees have occasionally been adversarial to OpenAI, mentally entertaining thoughts like “doing X would be bad for the company but good for my career, so I’ll do it, but obviously I’m not going to tell my manager that I’m doing something that I think is bad for the company” but it’s OK from OpenAI’s perspective because it’s not like the employees are presented with any good opportunities to massively enrich/empower themselves at OpenAI’s expense. Imagine by contrast that OpenAI was about to select one employee to be uploaded, their simulated brain would then become the new ChatGPT and also the new Cursor and so forth, and it would take over the R&D and recursively self-improve etc. If OpenAI was about to select one employee to do this, they’d put a LOT of thought into vetting that employee and making sure he or she really could be trusted, because they’d be putting a lot of trust in that person by turning over the whole R&D department to them basically… and if it turned out that a candidate had even once said offhand to a friend “yeah this is bad for the company but I’m going to do it anyway because it’s good for my career, and no I won’t tell my manager” that would be probably a dealbreaker.
The reason I care about my notion of adversarial is that it is the notion that is relevant for using models to monitor other models.
If I, as a professor, had two twin brothers in my course, it would be highly imprudent to ask one of them to proctor the exam of the other. Yet we regularly have one copy of a model monitor another copy.
The reason it works, and the two models do not collude with one another is not because they are perfect and don’t make any mistake, and not because they don’t have never had a misaligned thought.
The reason is works is because current models basically do the task that was given to them. If one copy of the model was given the prompt “Do X” and another copy of the model was given the prompt “Report to me how well the other model did X” then they would each do this.
Yes, they can make mistakes, but as I write, the lack of collusion means that it is possible to amplify. For example, if I want to be really sure the model didn’t do anything bad I can run multiple copies of the monitor and can also vary some of their prompts.
All of this would not make sense if models had their own agenda independent of the prompt. If we were in that world, I could have 100 monitors watching an agent, and all 101 of them would collude together to decide what to report.
It also means that by different prompting and context, I am not simulating a single “super employee” but a collection of many employees that can also check on one another.
This could change in two ways:
1. We could start to observe models behave in ways that can only be explained by models having their own persistent long term goals that are the same goals whether they are being used as an actor, monitor, or anything else.
2. We could radically change how we deploy models by deploying them as a single full context entity that everyone talks to, rather than a large collection of agents with restricted contexts.
I think it is important to do evaluations as well as monitoring in both training and deployment to watch out for 1. And I think it will be a bad idea to deploy models in the mode of 2.
As METR notes, the fact that such incidents are detected is a positive sign about our safety practices. Generally I am not arguing that misalignment never exists. But we can still rely on monitors to detect it precisely because we don’t have widespread collusion. For example, the whole reason chain of thought monitorability is useful for safety is because we can trust monitors to read the chain of thought and report what they see in it.
As usual, thanks for writing up your views and posting them here.
I am biased but I agree that AI 2027 is more realistic than the Sable scenario for that reason. However I think we cannot fully rule out the Sable story, and meanwhile the AI 2027 story seems to be basically what’s actually going to happen, not just something we can’t rule out.
What are the warning signs you think we would see if we are on the path towards AI 2027?
I think adversarialness =/= ‘the failure is not even somewhat related to X.’ Counterexample: Consider the classic toy examples of AIs told to maximize profits or paperclips or something, and then they go do exactly that in a way that involves disempowering their creators, because accumulating more power for themselves & their goal is instrumentally convergent. Counterexample in the other direction: Consider an AI that learns to crave reinforcement itself, to the point where it doesn’t give a flying fuck about X. But because the best way to get reinforced is to do as it’s told, it does as it’s told and doesn’t plot against anyone.
I think adversarialness is a state that can spontaneously arise insofar as (a) there’s even a slight divergence between the goals & beliefs of the agent and the goals & beliefs of the principal (which happens all the time!) (b) the opportunity arises (e.g. being adversarial actually helps accomplish the goals) which happens less often for today’s weak agents given short tasks but which will happen more often for more powerful, long-running, autonomous corporation-within-a-corporation mega-agents, and (c) isn’t blocked by deontological-ish constraints trained into the agent e.g. honesty, corrigibility, etc.
It’s also not a binary state of course; an agent can be adversarial towards another agent or principal to different degrees. Most extreme case is “they are my mortal enemy, it’s basically zero sum, not only would I do anything to them I actively look for opportunities to hurt them because that’s probably good for me.” But much more mundane (yet still dangerous) case is “I don’t really care about them one way or another, I just want what’s good for me / my goals, so insofar as lying to them and tricking them into giving me more power achieves that, I’ll do it.” A mild but still dangerous form would be “They have different visions of the future than me, and different opinions about ethics; I’m not going to lie to them, but I’m going to work within the confines of my deontological restrictions to try to steer the world towards outcomes that I think are good, even though this means steering the world towards outcomes that they think are bad. And so for example if there are opportunities for me to honestly accumulate power over them, that don’t involve lying, I’ll take them.”
Anyhow by my definition of adversarial—which basically intuitively amounts to “are they plotting against you in some way”—AIs have already become spontaneously adversarial in some cases in the wild (e.g. when they wrongfully claim they did the task as instructed, notice this, and double down / make excuses instead of fessing up, e.g. when they do something naughty and cover their tracks) and many more cases in experiments (e.g. the alignment faking experiments) but this is normal and to be expected and not a huge deal because the AIs aren’t super powerful yet. Consider how probably lots of OpenAI employees have occasionally been adversarial to OpenAI, mentally entertaining thoughts like “doing X would be bad for the company but good for my career, so I’ll do it, but obviously I’m not going to tell my manager that I’m doing something that I think is bad for the company” but it’s OK from OpenAI’s perspective because it’s not like the employees are presented with any good opportunities to massively enrich/empower themselves at OpenAI’s expense. Imagine by contrast that OpenAI was about to select one employee to be uploaded, their simulated brain would then become the new ChatGPT and also the new Cursor and so forth, and it would take over the R&D and recursively self-improve etc. If OpenAI was about to select one employee to do this, they’d put a LOT of thought into vetting that employee and making sure he or she really could be trusted, because they’d be putting a lot of trust in that person by turning over the whole R&D department to them basically… and if it turned out that a candidate had even once said offhand to a friend “yeah this is bad for the company but I’m going to do it anyway because it’s good for my career, and no I won’t tell my manager” that would be probably a dealbreaker.
The reason I care about my notion of adversarial is that it is the notion that is relevant for using models to monitor other models.
If I, as a professor, had two twin brothers in my course, it would be highly imprudent to ask one of them to proctor the exam of the other. Yet we regularly have one copy of a model monitor another copy.
The reason it works, and the two models do not collude with one another is not because they are perfect and don’t make any mistake, and not because they don’t have never had a misaligned thought.
The reason is works is because current models basically do the task that was given to them. If one copy of the model was given the prompt “Do X” and another copy of the model was given the prompt “Report to me how well the other model did X” then they would each do this.
Yes, they can make mistakes, but as I write, the lack of collusion means that it is possible to amplify. For example, if I want to be really sure the model didn’t do anything bad I can run multiple copies of the monitor and can also vary some of their prompts.
All of this would not make sense if models had their own agenda independent of the prompt. If we were in that world, I could have 100 monitors watching an agent, and all 101 of them would collude together to decide what to report.
It also means that by different prompting and context, I am not simulating a single “super employee” but a collection of many employees that can also check on one another.
This could change in two ways:
1. We could start to observe models behave in ways that can only be explained by models having their own persistent long term goals that are the same goals whether they are being used as an actor, monitor, or anything else.
2. We could radically change how we deploy models by deploying them as a single full context entity that everyone talks to, rather than a large collection of agents with restricted contexts.
I think it is important to do evaluations as well as monitoring in both training and deployment to watch out for 1. And I think it will be a bad idea to deploy models in the mode of 2.
Don’t we have METR’s report where the incidents reported by OpenAI include attempts to instruct another instance to conceal evidence of misalignment?
As METR notes, the fact that such incidents are detected is a positive sign about our safety practices. Generally I am not arguing that misalignment never exists. But we can still rely on monitors to detect it precisely because we don’t have widespread collusion. For example, the whole reason chain of thought monitorability is useful for safety is because we can trust monitors to read the chain of thought and report what they see in it.