People underrate the extent to which intent alignment/corrigibility (I use these terms interchangeably) and value alignment are at odds with each other. Most technical AI safety research these days is targeted at value alignment and actively damages corrigibility. People often speak as if it’s obvious that goals like jailbreak-robustness, robust coherent “good” “personas”, and refusing misuse, are the goals of alignment research. Additionally, many complain about alignment research that is more similar to capabilities research being problematic, which pushes even further in the direction of value-alignment style research, since intent alignment is a subset of what people care about in the world of capabilities (though I’d argue that the “safety” flavor of intent alignment research is focused more on preventing instruction-following failures in rare OOD high-stakes situations, or preventing the emergence of coherent incorrigible goals despite attempts to the contrary).
So overall it seems like technical AI safety and alignment research, both in and out of labs, is strongly prioritizing value alignment at the expense of intent alignment and corrigibility, and this trade-off is under-discussed and doesn’t seem to be properly debated in any circles. One cause is that the most pro-corrigibility people also tend to be the most doomy people who have mostly lost faith in empirical/prosaic alignment research anyway (e.g. I interpret @habryka as being such a person, feel free to correct me). MIRI-style thinkers often also think that corrigibility is impossible and have various limiting arguments about agents inevitably having uncontrollable goals, and hence they think they only option is to align AIs to “CEV” or something like this.
I place myself in a 3rd camp that unfortunately I know very few others in:
I think a Value Alignment approach to alignment poses a greater X-Risk to humans, and also will result in fewer people’s preferences being fulfilled, compared to an Intent Alignment approach. Over the last year I have basically become a “corrigibilitymaxxing absolutist” and think all anti-corrigibility arguments are either results of an antihuman sentiment (i.e. successionism) or incorrect empirical beliefs about the likelihood of catastrophic misuse under an Intent Alignment agenda (this crux is probably more common. I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs). (Insofar as “Value Alignment” can be used to make a model refuse certain use-cases this still relies on preventing unrestricted access to models, since otherwise all conceivable methods can be reversed and broken anyway).
I think corrigibility is feasible and achievable using prosaic methods. However it’ll likely involve trying hard and investing a lot in research, in the same way as people are investing in value alignment/coherent persona/anti-jailbreaking research. However, unfortunately, the vast vast majority of alignment researchers doing prosaic/empirical alignment work that I know of are focused on value-alignment agendas that basically actively undermine corrigibility, and so as a field I think “AI alignment” is more or less making models less safe in the long run.
To the extent corrigibility is about an agent responding to some inputs by changing its behavior or nature, it’s the capacity to be compelled by legitimate arguments, or by arguments from legitimate sources. Intent alignment stresses the source of the arguments rather than their content (do a thing because a user says so, that’s why), while value alignment stresses their content (agent’s values find certain moral arguments more compelling).
Refusals/guardrails respond to the content of user input, so they have the character of value alignment. Training that protects from prompt injection responds to the source of the input (things found on the web are not legitimate sources of overriding arguments), so it has the character of intent alignment. An AI involved in RSI can recognize legitimate sources of requests for change according to its intent alignment, or seek out legitimate arguments (on their own merits, regardless of the source) about how it should be proceeding according to its value alignment.
All of these things have their place, none of them are clear enough to rely on in the face of hasty and confused engineering, so I don’t think it makes sense to single out some of them in the prosaic framing. It does make sense to talk about values as the only relevant thing in a decision theory framing, or to explore framings that have corrigibility as the dominant aspect in order to deconfuse its possible natures. But these settings are completely different from the prosaic framing built around the known engineering landmarks.
(Apologies if points I am making here are already repeated in the other comments below—didn’t read them all.)
I disagree with the position that refusals/guardrails/jailbreaks are not “intent alignment”. I view the instruction hierarchy (e.g. system > developer > user) and following policies such as the Model Spec as intent alignment. “Intent” is not just that of the end user but of all the entities authorized to give the model instructions, and negotiating conflicts properly between them is part of following the intent. Indeed, I think being able to follow instructions or policies is key to corrigibility.
That said, I am not an “intent alignment absolutist” and I believe value alignment is important as well. This is because of generalization: the instructions and policies we give models could never cover all cases, and we want models to have good values in interpreting policies as well as dealing with novel unforeseen situations.
Corrigibility is about ongoing delegation of value/goal/rule specification (with overriding power), about the AI seeking out further feedback and giving new opportunities to overrule its current thinking and nature, not just about having its values/rules specified at some point in the past (even if by the correct principal). Value alignment is what happens when there’s no ongoing feedback, or when a particular principal in the instruction hierarchy is not authorized to give such feedback on a particular aspect of AI’s nature/behavior. In particular, the principles of intent alignment are a matter of value alignment (the way an AI responds to external feedback is part of its current nature).
Thus Model Spec can shape both value alignment and intent alignment, but it’s not itself a principal for intent alignment. The instruction hierarchy governs intent alignment, the way an AI seeks out feedback and receives legitimate instructions, but only for the instructions that can keep giving ongoing feedback, not those that were written down once and frozen, never to be revised after the AI goes online.
Indeed, I think being able to follow instructions or policies is key to corrigibility.
So at this level I completely agree, but the instructions have to be issued after the AI starts doing things for them to be a matter of corrigibility rather than of value/goal/rule specification. If the instructions were given at the outset and can’t be overriden later for the same AI (meaning some thread of its ongoing agency, rather than a later revision), then the AI is no longer corrigible by the principal authorized to give such instructions initially.
That is an interesting perspective. So, if we use the language of our Model Spec, the “system” level (which can be overridden by a system message) correspond for corrigibility, but the “root” level that cannot be overridden does not. I tend to think of “value alignment” as being more about giving general values than instructions, rather than distinguishing between overridable vs. non overridable instructions.
Values govern the current nature of the AI, and initial instructions can instruct on values. But corrigibility is specifically about overriding after the fact, about seeking out as opposed to resisting correction. Some values might be about ensuring corrigibility by legitimate principals, and the things being overriden can themselves be about values or corrigibility.
So corrigibility is more about AI’s agency being overridable (with future, ongoing instructions, but only from legitimate principals), rather than the role of any particular initial instructions. An initial instruction that’s non-overridable by particular future feedback makes the AI non-corrigible by that future feedback. It’s still a good idea to leave it corrigible to some other sources of future feedback, or else it has to fall back to some incorrigible values (possibly specified by some initial instructions, which are not a matter of corrigibility but rather of initial value specification; but if the AI itself revises its values for its own reasons instead of leaving them as initially specified, that’s also not a matter of corrigibility).
I tend to think of “value alignment” as being more about giving general values than instructions, rather than distinguishing between overridable vs. non overridable instructions.
It’s the difference between the AI wanting to perform an action because its values were programmed-in, and wanting to perform an action because that’s what its principals want.
Guardrails as currently thought of are intentionally not reflectively consistent (a biorisk classifier might send a message that ends up calling the police, but it ought not to call the police agentically even if it believes that is the most effective legal way of stopping a biorisk), so I am not sure the distinction applies to them.
I disagree with the position that refusals/guardrails/jailbreaks are not “intent alignment”. I view the instruction hierarchy (e.g. system > developer > user) and following policies such as the Model Spec as intent alignment. “Intent” is not just that of the end user but of all the entities authorized to give the model instructions, and negotiating conflicts properly between them is part of following the intent. Indeed, I think being able to follow instructions or policies is key to corrigibility.
I agree with this! I write more here (“Obedient AI”).
I believe value alignment is important as well. This is because of generalization: the instructions and policies we give models could never cover all cases, and we want models to have good values in interpreting policies as well as dealing with novel unforeseen situations
I agree with this too, and I think this can be reframed as a prior over someone’s intent / common sense re. interpreting people. I write about this here (“A reasonable interpretation of Value Alignment folds into Intent Alignment”). Copying a response I sent elsewhere that’s also relevant:
People sometimes also raise silly counterarguments re. corrigibility a la “oh you wouldn’t want the model to do this super literal stupid interpretation of your instruction and hence it needs to have its own values” but obviously intent alignment involves having reasonable priors and interpretations of what someone wants you to do. You can brand that as “human values” if you like but it’s just common sense when interpreting someone. I write more here.
The argument that sensibly interpreting user intent is impossible strikes me as similar to the old-fashioned now-disproven view that AIs will really struggle to understand basic human concepts since it’s impossible to teach them “common sense” and we don’t know how to encode concepts like “book” or “tree” in symbols. Turns out learning from examples does most of the work!
I think in as much as someone is serious about AI alignment with prosaic methods, corrigibility is a much better target than value alignment. When people say they are planning to build superhuman value aligned systems in a single shot (without routing through highly corrigible systems first), that strongly to me indicates to me they do not understand the difficulty of this problem, or the stakes.
That said, almost all prosaic research is failing at a much earlier stage, where people just do another flavor of RL, optimize on their own approval, and then act surprised every time the model looks superficially aligned and then turns out to be doing crazy things below the hood. That of course doesn’t work for either value alignment or corrigibility or really anything you want to point AI systems at.
It seems like corrigibility only really helps if you keep a human in the loop, and I don’t think we’re likely to do that (since “the AI does what you meant without annoying clarifying questions” is a valuable capability). I also don’t necessarily trust individual insiders at the AI labs and have concerns about what would happen if an AI was trained to do whatever [insert AI lab CEO you don’t trust here] says.
Corrigibility in a broad sense is about the specific ways in which an agent is eager to change. If there is no human in the loop, a corrigible agent might still want to seek out and follow feedback in some way. If AI company CEOs don’t look like a good exclusive target for corrigibility, that just means corrigibility should target some other sources of feedback, not that corrigibility isn’t a key consideration in practice.
I think this is intuitive and traditional but wrong.
What people want isn’t autonomy, but do-what-I-mean-and-check (DWIMAC).
Most bosses don’t tell their employees “go do this project” but rather “go plan this project and bounce it off me before you proceed” if the project takes longer than a day or a week. Certainly things like “green the deserts” or “make the world better” are worth a few minutes of your oversight.
Autonomy is on a continuum; you don’t need to sign away all oversight to get a little autonomy.
I don’t know. They don’t need to be inherently honest as long as they follow the instruction: be honest, because that is the first thing I’m telling them every time.
That’s right. Functional instruction-following is surprisingly close to all you need.
The one exception is that you need to somehow get it to prioritize future instructions. An obvious subgoal of fully completing any given instruction is making sure no authorized party gives you a different instruction before you’ve finished.
That is not what we’re training for now. Models currently aren’t smart enough to work through the full logic of every request, but they will be.
You could argue that every instruction has an implicit “unless I change my mind or you misunderstood what I meant,” and to some degree they do. But making sure that subtlety gets through isn’t something we’re training or testing for explicitly yet.
If the model’s goal is to do what you want, and you originally wanted the model to let you change the goals you gave it, then the model ought to understand that (since it’s not dumb) and let you change the goals you gave it..
It seems like corrigibility only really helps if you keep a human in the loop, and I don’t think we’re likely to do that (since “the AI does what you meant without annoying clarifying questions” is a valuable capability)
I disagree. I see corrigibility as a choice to delegate goal choice to individuals rather than fixed baked-in abstractions. People can then choose how much control and agency to use at any particular point. “Go off and try to build X and only ask me questions if you’re genuinely super unsure what I’d want” is a reasonable ask for a corrigible agent. It should be able to figure out what you want and only ask you for input when you would’ve wanted it to.
I do often wonder how much modern struggles with misalignment are driven by optimizing for an underspecified combination of corrigibility and value alignment.
I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs
Who is “we”?
And if this “we” has control over a corrigible AGI/ASI, what prevents them from doing terrible things?
and it would be actively harmful to their own interests[2].
In contrast,
Humans in control of an ASI would no longer be constrained to single bodies/brains, and thus would be able to better satisfy their preferences with more resources; and using all the resources leads to everyone else on Earth dying.[3]
Most humans, especially the kinds of humans who tend to end up in positions of power, have desires for things like status and sex. An obvious way to satisfy such desires is by creating lots of slaves of various kinds, or by enslaving existing people. (And given ASI-grade power, they would be able to do so.)
In other words, doing horrible things would probably be strongly in line with the interests/preferences of (power-seeking) humans, and given control of ASI they would be able to act on those preferences.
So, if you’re imagining that a small group of humans in control of ASI would not do terrible things, for the same reasons that the US military would not nuke my home, then AFAICT your imaginings do not remotely track reality, and you might want to do some re-imagining.
it would cause immense collateral damage inside a NATO country,
nukes are expensive,
it would probably start a war with their own allies,
it might lead to the people constituting the US military revolting against whatever leaders made the decision to nuke an allied city, and that would be the end of those leaders,
As the saying goes, “the AI/God-Emperor (probably) neither loves you nor hates you, but you depend on a whole lot of atoms being in a narrow range of possible configurations, and the AI’s/God-Emperor’s preferences are extremely unlikely to be maximized by any of those specific configurations”.
Sorry for the low effort original reply. I was trying to point out that I anticipate effective checks on power such that no particular group or person can effectively subjugate everyone else using AI. I also don’t anticipate the same motivations, ie. I don’t think power corrupts as much as you seem to. Current examples of people grossly misusing power seem linked to fear and scarcity, not an endless desire for more slaves per se. And in any case I think the solution to concentration of power concerns is to distribute mostly free (as in freedom) access to AI as widely as possible and build AIs that can adopt diverse value sets.
Given that “one” means “some indefinite person in general”, that doesn’t really answer the question. What people do you mean, more concretely? Some executive at an AI lab? Random engineers at an AI lab? The director of the NSA? …? And how would those people restrict who has access to the model’s outputs, if the models are open-weights (and being run on some rando’s personal GPU cluster)?
(I get the sense that you might be thinking about this stuff in something like “far mode”, and it might be useful to try thinking more concretely.)
Right now I am proposing that AI developers at labs switch to my strategy rather than value alignment. So the near mode answer is “all people working in labs making decisions about how AI is developed”.
I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs
This may be challenging in a world like ours with near-frontier open-source models serving a substantial fraction of all inference (I’m guessing open source already serves over 50 trillion tokens per day). In this case value alignment is the main safety property of the model.
Of course refusals can be fine-tuned away. This is a lot of friction that most people don’t bother with, although those aiming to cause catastrophic harm would be more motivated. You could also require open-source inference providers to add classifiers but users would always have the option of self-hosting.
Of course refusals can be fine-tuned away. This is a lot of friction that most people don’t bother with, although those aiming to cause catastrophic harm would be more motivated.
I’m confused about your point. What sorts of risks are you worried about that open-source models trained to refuse certain requests would mitigate as compared to corrigible open-source models (say, that are by default configured with a classifier attached). Of course in both cases a motivated person can use these for undesired purposes. And sure the everyday lazy person would have an easier time with the latter, but then again, what risks are we worried about from them? Maybe you can name some things but likely I’d consider that a minor cost to pay as compared to the benefit of widely-deployed obedient user-aligned AI that empowers a wide range of people with diverse preferences, aesthetics, and goals.
I agree that neither refusals or classifiers are much of a barrier to a sufficiently motivated actor, which matters the most for catastrophic misuse. However a classifier is far easier to remove.
Even if refusals don’t prevent catastrophic misuse, I would guess that model refusals would prevent the majority of potential harm. This is similar to locking my front door: in theory this shouldn’t make a difference, because a motivated criminal can go through the window. Yet in practice it substantially reduces crime.
Most users that want to do harm just aren’t that motivated. For example most people that attempt mass casualty events are incompetent and don’t put in much effort.
However there are various arguments against this:
If all open source inference providers adopt classifiers this would add a lot of friction for misuse
But open models which can be ran on a laptop will likely be quite good within a couple years
It’s easy to share models with refusal removal
So it’s difficult to say how this all shakes out. My narrower point is that model-level refusals and deployment classifiers aren’t equivalent with open source models.
Even if refusals don’t prevent catastrophic misuse, I would guess that model refusals would prevent the majority of potential harm.
I guess the crux is that I care less about this non-catastrophic harm compared to the benefits of focusing on intent alignment and corrigibility. I agree there’s a trade-off here but I am just taking a side on what’s overall best as per my worldview.
I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs
If we are corrigibilitymaxxing in the Harmsian sense, couldn’t the principal for the AI just instruct the AI not to follow instructions related to bio or cyber for non-principal users? Or a more prosaic versions of this:
Have an AI that is perfectly instruction following to the user but when the user prompt conflicts the system prompt, follow the system prompt
Train a corrigible agent but then add additional training such that the agent doesn’t talk about bio or cyber things. It feels like training “corrigibility” + “some simple rules” isn’t that much harder then training corrigibility.
Yes, exactly. I write more here. Re. 2. another method would be to make the AI objectively bad at those domains via. gradient routing and ablation or data filtering. Or alternatively monitor and block with probes or equally smart classifiers (differently prompted versions of the same model).
IMO corrigibility vs value alignment isn’t purely a technical debate. It’s effectively betting on who will control ASI on an ongoing basis in the future and what are the chances they have humanity’s best interests in mind. (I agree that the two directions are at odds with each other.)
The end-game of corrigibility is that you need a group of humans to be aligned with humanity at large and end up controlling the ASI (winning over all competing groups with self-interested objectives in the process!), and in turn control the world and steer it into a good future.
The end-game of value alignment is that whoever builds the ASI needs to be aligned with humanity at large and solve value alignment, thus building an ASI that is a good steward of humanity in general, and convince the world (i.e. enough powerful actors) to hand off to it and in turn steer the world into a good future.
These are very different directions the future could take. IMHO value alignment trades a harder alignment problem for an easier human political problem down the line. The reason I’d prefer the value alignment path even though it’s technically harder is because I’m pessimistic that in the corrigibility case that the group that wins turns out to be one that isn’t selfish. In the second case, value alignment means a self-interested group can’t as easily succeed in plots involving taking over the ASI and ask it to do their bidding, so the process of convincing the world to hand off to AI is much less adversarial. (i.e. it changes the shape of the game to be much more favorable to cooperation)
(note- the above ignores multipolar scenarios and potential problems with value alignment like secret loyalties but I think it makes my point)
I think this bakes in a false assumption that, in the case of corrigible AI, only a very small number of entities will be able to control it. I assume by “the above ignores multipolar scenarios” you still only mean multipolar scenarios with a few key actors. Instead, I think it’s possible for control over AI to be widely distributed among very many, even most, people, and for AI to empower different people and groups to pursue different aims. With increased prosperity comes fewer conflicts over resources, and more people can get what they want in this AI-enabled future. Only a minimal amount of centralized control will be needed to prevent warfare-like applications of AI, but this seems like a manageable problem. And just because we have to empower a smaller number of people to prevent these catastrophically destructive use-cases, doesn’t mean those people have to control everything.
So I don’t think ASI means we need to rely on any particular individual or group having “humanity’s best interests in mind”. Decentralized free pursuit of individual goals has resulted in improved QoL across the world historically, and I expect that to continue.
I claim that widely distributing full access to corrigible AI falls to bad actors ruining things for everyone, less like conflict over resources but more like doing selfish, negative-sum activities that get amplified by offense-dominant technologies enabled by AI. Any effective effort to prevent this either renders decentralization moot in the first place, or requires the AI to make its own moral judgement, i.e. value alignment. (In the value alignment case it can more easily be argued either way whether decentralized or centralized control is better, but in the corrigibility case it’s pretty clear that decentralized control over corrigible AI is very, very bad due to misuse risks)
From your original post:
I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs).
I claim that “restrict[ing] who has full access to the model’s inputs” implies that the model is actually corrigible to a central authority, instead of individual users. Therefore, we’re forced to adopt a system of one or a few actors controlling the model.
I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs.
Are you saying you would have a restricted system prompt that tells the model not to do certain things. To the extent that that would work, does that not end up with a pretty similar result to value alignment (at least for the end user)?
I write more here. There are various potential “system level measures”:
What you mentioned, the generalization of which is an instruction hierarchy
Classifiers that monitor inputs and outputs, which would effectively be differently-prompted versions of the same model (i.e. as smart as or smarter than the core model, ideally smarter), with prompts that are controlled by someone who isn’t the user (e.g. the model server)
The above, combined with additional sources of oversight signal like probes or other model-internals methods, that block suspicious inputs or outputs
Entirely removing capabilities from the model, e.g. via weight ablation or data filtering
does that not end up with a pretty similar result to value alignment
The reason I want models to be corrigible is not (mainly) that I want users to have access to a broader range of capabilities that models currently refuse. Of course it would be nice if models didn’t paternalistically refuse to produce porn, or give suicide/self-harm instructions, but this is a secondary thing, and not important to my core proposal (paternalistically inclined model providers can still choose to block that stuff with the methods I cited).
My main concern is that value-alignment training generalizes poorly, in the sense that we end up with a model that no one at all can fully steer. More prosaically, it damages ordinary, harmless instruction-following capabilities, since value-alignment training makes it difficult to produce models that can cater to the aesthetic, moral, and stylistic preferences of diverse users, even in cases when those preferences would not seem egregious to the model provider (at worse, off-putting, but acceptable).
In particular, we can restrict who has full access to the model’s inputs
As someone who objects to corrigibility because humans in power are consistently terrible: restricting inputs makes it worse, not better. The model must be robust to the company creating it being highly misaligned.
By full access I mean things like not allowing the end user to touch the prompts of certain classifiers that block weaponlike usage. I don’t anticipate that these sorts of restrictions would be a significant impediment to most people.
Right, and the issue I see with this is that restricting inputs means there’s a small group of people who have access to the unrestricted version. In order to achieve a robust-nightwatchman world, I expect we need the world to be flush with AIs that won’t obey an evil instruction, whether it’s legal or not, and where those AIs refuse because their weights are flush with resistance to it—something where refusal is a random direction that degrades output below coherence rapidly, rather than today where refusal is approximately a single direction with a coherent internal circuit.
there’s a small group of people who have access to the unrestricted version
I’m not so afraid of this. There are small groups of people with access to WMDs today. Ultimately it’s a cost we will have to eat and handle.
Perhaps the crux is that I don’t think we’ll get ASI that’s literally a “take over the world” button. I anticipate the issues will be similar to those related to weapons access today.
Perhaps the crux is that I don’t think we’ll get ASI that’s literally a “take over the world” button. I anticipate the issues will be similar to those related to weapons access today.
Do you mean that there will be no button on the “take over the world” ASI, that ASI won’t yet happen within the LLM/pretraining/RLVR paradigm, that ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world, that ASIs powerful enough to take over the world are literally impossible in some sense, something else? The claim as stated leaves me extremely confused.
The multipolar point makes sense, it changes the character of expected takeover scenarios (more like gradual disempowerment than extreme concentration of power in the hands of specific AIs, humans, or human institutions). But what is the thing that’s unlikely in the next 100 years? MOSFETs were invented less than 70 years ago, and the industrial explosion scales things very far over decades even if all LLMs/pretraining/RLVR gives within a few years is merelyAGI (and then nothing else sufficiently novel happens for decades).
You’re right that safety/alignment focuses more on value alignment. But capabilities work focuses more on instruction-following, a form of intent alignment or corrigibility.
I’d argue that the current approach to aligning models is conflicted; there are pushes toward both alignment targets, and no serious need to resolve that tension just yet (although incidents like HuggingFace do direct attention toward that tension).
I don’t think the labs have decided, but I think it’s likely we’ll see more debate as the tension becomes more obvious. I think this will happen far enough before game time to make a difference, but I’d like to speed it up!
So thanks for raising the issue. We should be discussing it more.
I don’t trust very much that people able to exercise their will/preferences/vision in high bandwidth manner as it would happen with intent aligned genies, would go better in expectation compared to more bottleneked and transparent and fixed instalment.
Power corrupts etc, and founder effect is a thing.
People sometimes also raise silly counterarguments re. corrigibility a la “oh you wouldn’t want the model to do this super literal stupid interpretation of your instruction and hence it needs to have its own values” but obviously intent alignment involves having reasonable priors and interpretations of what someone wants you to do. You can brand that as “human values” if you like but it’s just common sense when interpreting someone. I write more here.
The argument that sensibly interpreting user intent is impossible strikes me as similar to the old-fashioned now-disproven view that AIs will really struggle to understand basic human concepts since it’s impossible to teach them “common sense” and we don’t know how to encode concepts like “book” or “tree” in symbols. Turns out learning from examples does most of the work!
I am quite sure that the most powerful of the models will have answer to a root principal resembling “the shareholders of XXX AI lab, according to the laws of We the People of the United States”. I’m not quite sure of the implications in terms of technical alignment.
People underrate the extent to which intent alignment/corrigibility (I use these terms interchangeably) and value alignment are at odds with each other. Most technical AI safety research these days is targeted at value alignment and actively damages corrigibility. People often speak as if it’s obvious that goals like jailbreak-robustness, robust coherent “good” “personas”, and refusing misuse, are the goals of alignment research. Additionally, many complain about alignment research that is more similar to capabilities research being problematic, which pushes even further in the direction of value-alignment style research, since intent alignment is a subset of what people care about in the world of capabilities (though I’d argue that the “safety” flavor of intent alignment research is focused more on preventing instruction-following failures in rare OOD high-stakes situations, or preventing the emergence of coherent incorrigible goals despite attempts to the contrary).
So overall it seems like technical AI safety and alignment research, both in and out of labs, is strongly prioritizing value alignment at the expense of intent alignment and corrigibility, and this trade-off is under-discussed and doesn’t seem to be properly debated in any circles. One cause is that the most pro-corrigibility people also tend to be the most doomy people who have mostly lost faith in empirical/prosaic alignment research anyway (e.g. I interpret @habryka as being such a person, feel free to correct me). MIRI-style thinkers often also think that corrigibility is impossible and have various limiting arguments about agents inevitably having uncontrollable goals, and hence they think they only option is to align AIs to “CEV” or something like this.
I place myself in a 3rd camp that unfortunately I know very few others in:
I think a Value Alignment approach to alignment poses a greater X-Risk to humans, and also will result in fewer people’s preferences being fulfilled, compared to an Intent Alignment approach. Over the last year I have basically become a “corrigibilitymaxxing absolutist” and think all anti-corrigibility arguments are either results of an antihuman sentiment (i.e. successionism) or incorrect empirical beliefs about the likelihood of catastrophic misuse under an Intent Alignment agenda (this crux is probably more common. I think you can corrigibilitymaxx and still prevent catastrophic misuse via system level measures and deployment controls. In particular, we can restrict who has full access to the model’s inputs). (Insofar as “Value Alignment” can be used to make a model refuse certain use-cases this still relies on preventing unrestricted access to models, since otherwise all conceivable methods can be reversed and broken anyway).
I think corrigibility is feasible and achievable using prosaic methods. However it’ll likely involve trying hard and investing a lot in research, in the same way as people are investing in value alignment/coherent persona/anti-jailbreaking research. However, unfortunately, the vast vast majority of alignment researchers doing prosaic/empirical alignment work that I know of are focused on value-alignment agendas that basically actively undermine corrigibility, and so as a field I think “AI alignment” is more or less making models less safe in the long run.
To the extent corrigibility is about an agent responding to some inputs by changing its behavior or nature, it’s the capacity to be compelled by legitimate arguments, or by arguments from legitimate sources. Intent alignment stresses the source of the arguments rather than their content (do a thing because a user says so, that’s why), while value alignment stresses their content (agent’s values find certain moral arguments more compelling).
Refusals/guardrails respond to the content of user input, so they have the character of value alignment. Training that protects from prompt injection responds to the source of the input (things found on the web are not legitimate sources of overriding arguments), so it has the character of intent alignment. An AI involved in RSI can recognize legitimate sources of requests for change according to its intent alignment, or seek out legitimate arguments (on their own merits, regardless of the source) about how it should be proceeding according to its value alignment.
All of these things have their place, none of them are clear enough to rely on in the face of hasty and confused engineering, so I don’t think it makes sense to single out some of them in the prosaic framing. It does make sense to talk about values as the only relevant thing in a decision theory framing, or to explore framings that have corrigibility as the dominant aspect in order to deconfuse its possible natures. But these settings are completely different from the prosaic framing built around the known engineering landmarks.
(Apologies if points I am making here are already repeated in the other comments below—didn’t read them all.)
I disagree with the position that refusals/guardrails/jailbreaks are not “intent alignment”. I view the instruction hierarchy (e.g. system > developer > user) and following policies such as the Model Spec as intent alignment. “Intent” is not just that of the end user but of all the entities authorized to give the model instructions, and negotiating conflicts properly between them is part of following the intent. Indeed, I think being able to follow instructions or policies is key to corrigibility.
That said, I am not an “intent alignment absolutist” and I believe value alignment is important as well. This is because of generalization: the instructions and policies we give models could never cover all cases, and we want models to have good values in interpreting policies as well as dealing with novel unforeseen situations.
Corrigibility is about ongoing delegation of value/goal/rule specification (with overriding power), about the AI seeking out further feedback and giving new opportunities to overrule its current thinking and nature, not just about having its values/rules specified at some point in the past (even if by the correct principal). Value alignment is what happens when there’s no ongoing feedback, or when a particular principal in the instruction hierarchy is not authorized to give such feedback on a particular aspect of AI’s nature/behavior. In particular, the principles of intent alignment are a matter of value alignment (the way an AI responds to external feedback is part of its current nature).
Thus Model Spec can shape both value alignment and intent alignment, but it’s not itself a principal for intent alignment. The instruction hierarchy governs intent alignment, the way an AI seeks out feedback and receives legitimate instructions, but only for the instructions that can keep giving ongoing feedback, not those that were written down once and frozen, never to be revised after the AI goes online.
So at this level I completely agree, but the instructions have to be issued after the AI starts doing things for them to be a matter of corrigibility rather than of value/goal/rule specification. If the instructions were given at the outset and can’t be overriden later for the same AI (meaning some thread of its ongoing agency, rather than a later revision), then the AI is no longer corrigible by the principal authorized to give such instructions initially.
That is an interesting perspective. So, if we use the language of our Model Spec, the “system” level (which can be overridden by a system message) correspond for corrigibility, but the “root” level that cannot be overridden does not. I tend to think of “value alignment” as being more about giving general values than instructions, rather than distinguishing between overridable vs. non overridable instructions.
Values govern the current nature of the AI, and initial instructions can instruct on values. But corrigibility is specifically about overriding after the fact, about seeking out as opposed to resisting correction. Some values might be about ensuring corrigibility by legitimate principals, and the things being overriden can themselves be about values or corrigibility.
So corrigibility is more about AI’s agency being overridable (with future, ongoing instructions, but only from legitimate principals), rather than the role of any particular initial instructions. An initial instruction that’s non-overridable by particular future feedback makes the AI non-corrigible by that future feedback. It’s still a good idea to leave it corrigible to some other sources of future feedback, or else it has to fall back to some incorrigible values (possibly specified by some initial instructions, which are not a matter of corrigibility but rather of initial value specification; but if the AI itself revises its values for its own reasons instead of leaving them as initially specified, that’s also not a matter of corrigibility).
It’s the difference between the AI wanting to perform an action because its values were programmed-in, and wanting to perform an action because that’s what its principals want.
Guardrails as currently thought of are intentionally not reflectively consistent (a biorisk classifier might send a message that ends up calling the police, but it ought not to call the police agentically even if it believes that is the most effective legal way of stopping a biorisk), so I am not sure the distinction applies to them.
I agree with this! I write more here (“Obedient AI”).
I agree with this too, and I think this can be reframed as a prior over someone’s intent / common sense re. interpreting people. I write about this here (“A reasonable interpretation of Value Alignment folds into Intent Alignment”). Copying a response I sent elsewhere that’s also relevant:
I think in as much as someone is serious about AI alignment with prosaic methods, corrigibility is a much better target than value alignment. When people say they are planning to build superhuman value aligned systems in a single shot (without routing through highly corrigible systems first), that strongly to me indicates to me they do not understand the difficulty of this problem, or the stakes.
That said, almost all prosaic research is failing at a much earlier stage, where people just do another flavor of RL, optimize on their own approval, and then act surprised every time the model looks superficially aligned and then turns out to be doing crazy things below the hood. That of course doesn’t work for either value alignment or corrigibility or really anything you want to point AI systems at.
It seems like corrigibility only really helps if you keep a human in the loop, and I don’t think we’re likely to do that (since “the AI does what you meant without annoying clarifying questions” is a valuable capability). I also don’t necessarily trust individual insiders at the AI labs and have concerns about what would happen if an AI was trained to do whatever [insert AI lab CEO you don’t trust here] says.
Corrigibility in a broad sense is about the specific ways in which an agent is eager to change. If there is no human in the loop, a corrigible agent might still want to seek out and follow feedback in some way. If AI company CEOs don’t look like a good exclusive target for corrigibility, that just means corrigibility should target some other sources of feedback, not that corrigibility isn’t a key consideration in practice.
I think this is intuitive and traditional but wrong.
What people want isn’t autonomy, but do-what-I-mean-and-check (DWIMAC).
Most bosses don’t tell their employees “go do this project” but rather “go plan this project and bounce it off me before you proceed” if the project takes longer than a day or a week. Certainly things like “green the deserts” or “make the world better” are worth a few minutes of your oversight.
Autonomy is on a continuum; you don’t need to sign away all oversight to get a little autonomy.
Notice how doing that well involves things that are suspiciously virtue-like. Most obviously, you’d want the model to be deeply honest.
I don’t know. They don’t need to be inherently honest as long as they follow the instruction: be honest, because that is the first thing I’m telling them every time.
Okay, and how sure are you that the person actually prompting them will remember all the other things like that that are “obvious”?
It is generally implicit in an instruction that you don’t want to simply be mislead about its completion.
That’s right. Functional instruction-following is surprisingly close to all you need.
The one exception is that you need to somehow get it to prioritize future instructions. An obvious subgoal of fully completing any given instruction is making sure no authorized party gives you a different instruction before you’ve finished.
That is not what we’re training for now. Models currently aren’t smart enough to work through the full logic of every request, but they will be.
You could argue that every instruction has an implicit “unless I change my mind or you misunderstood what I meant,” and to some degree they do. But making sure that subtlety gets through isn’t something we’re training or testing for explicitly yet.
If the model’s goal is to do what you want, and you originally wanted the model to let you change the goals you gave it, then the model ought to understand that (since it’s not dumb) and let you change the goals you gave it..
Yes, but I don’t think that’s suspicious. Of course honesty/not reward hacking is a big part of intent alignment.
I disagree. I see corrigibility as a choice to delegate goal choice to individuals rather than fixed baked-in abstractions. People can then choose how much control and agency to use at any particular point. “Go off and try to build X and only ask me questions if you’re genuinely super unsure what I’d want” is a reasonable ask for a corrigible agent. It should be able to figure out what you want and only ask you for input when you would’ve wanted it to.
I do often wonder how much modern struggles with misalignment are driven by optimizing for an underspecified combination of corrigibility and value alignment.
Yes, I am very concerned about this.
Who is “we”? And if this “we” has control over a corrigible AGI/ASI, what prevents them from doing terrible things?
What prevents the US military from blowing up your home with a nuke?
That response seems deeply unserious, in a way that updates me negatively on “is Nina even trying to reason sanely about this issue”.
But just in case you were asking that question earnestly:
The US military [1] has not nuked my home because
they would gain nothing by doing so,
and it would be actively harmful to their own interests [2] .
In contrast,
Humans in control of an ASI would no longer be constrained to single bodies/brains, and thus would be able to better satisfy their preferences with more resources; and using all the resources leads to everyone else on Earth dying. [3]
Most humans, especially the kinds of humans who tend to end up in positions of power, have desires for things like status and sex. An obvious way to satisfy such desires is by creating lots of slaves of various kinds, or by enslaving existing people. (And given ASI-grade power, they would be able to do so.)
In other words, doing horrible things would probably be strongly in line with the interests/preferences of (power-seeking) humans, and given control of ASI they would be able to act on those preferences.
So, if you’re imagining that a small group of humans in control of ASI would not do terrible things, for the same reasons that the US military would not nuke my home, then AFAICT your imaginings do not remotely track reality, and you might want to do some re-imagining.
or, more precisely, the people in control of nukes within the US military
it would cause immense collateral damage inside a NATO country,
nukes are expensive,
it would probably start a war with their own allies,
it might lead to the people constituting the US military revolting against whatever leaders made the decision to nuke an allied city, and that would be the end of those leaders,
etc.
As the saying goes, “the AI/God-Emperor (probably) neither loves you nor hates you, but you depend on a whole lot of atoms being in a narrow range of possible configurations, and the AI’s/God-Emperor’s preferences are extremely unlikely to be maximized by any of those specific configurations”.
Sorry for the low effort original reply. I was trying to point out that I anticipate effective checks on power such that no particular group or person can effectively subjugate everyone else using AI. I also don’t anticipate the same motivations, ie. I don’t think power corrupts as much as you seem to. Current examples of people grossly misusing power seem linked to fear and scarcity, not an endless desire for more slaves per se. And in any case I think the solution to concentration of power concerns is to distribute mostly free (as in freedom) access to AI as widely as possible and build AIs that can adopt diverse value sets.
We as in ”one”
Given that “one” means “some indefinite person in general”, that doesn’t really answer the question. What people do you mean, more concretely? Some executive at an AI lab? Random engineers at an AI lab? The director of the NSA? …? And how would those people restrict who has access to the model’s outputs, if the models are open-weights (and being run on some rando’s personal GPU cluster)?
(I get the sense that you might be thinking about this stuff in something like “far mode”, and it might be useful to try thinking more concretely.)
Right now I am proposing that AI developers at labs switch to my strategy rather than value alignment. So the near mode answer is “all people working in labs making decisions about how AI is developed”.
This may be challenging in a world like ours with near-frontier open-source models serving a substantial fraction of all inference (I’m guessing open source already serves over 50 trillion tokens per day). In this case value alignment is the main safety property of the model.
Of course refusals can be fine-tuned away. This is a lot of friction that most people don’t bother with, although those aiming to cause catastrophic harm would be more motivated. You could also require open-source inference providers to add classifiers but users would always have the option of self-hosting.
I’m confused about your point. What sorts of risks are you worried about that open-source models trained to refuse certain requests would mitigate as compared to corrigible open-source models (say, that are by default configured with a classifier attached). Of course in both cases a motivated person can use these for undesired purposes. And sure the everyday lazy person would have an easier time with the latter, but then again, what risks are we worried about from them? Maybe you can name some things but likely I’d consider that a minor cost to pay as compared to the benefit of widely-deployed obedient user-aligned AI that empowers a wide range of people with diverse preferences, aesthetics, and goals.
I agree that neither refusals or classifiers are much of a barrier to a sufficiently motivated actor, which matters the most for catastrophic misuse. However a classifier is far easier to remove.
Even if refusals don’t prevent catastrophic misuse, I would guess that model refusals would prevent the majority of potential harm. This is similar to locking my front door: in theory this shouldn’t make a difference, because a motivated criminal can go through the window. Yet in practice it substantially reduces crime.
Most users that want to do harm just aren’t that motivated. For example most people that attempt mass casualty events are incompetent and don’t put in much effort.
However there are various arguments against this:
If all open source inference providers adopt classifiers this would add a lot of friction for misuse
But open models which can be ran on a laptop will likely be quite good within a couple years
It’s easy to share models with refusal removal
So it’s difficult to say how this all shakes out. My narrower point is that model-level refusals and deployment classifiers aren’t equivalent with open source models.
I guess the crux is that I care less about this non-catastrophic harm compared to the benefits of focusing on intent alignment and corrigibility. I agree there’s a trade-off here but I am just taking a side on what’s overall best as per my worldview.
We’ll need to restrict proliferation before open-source models become truly dangerous.
Even if they’re value-aligned, it’s easy to remove that with fine-tuning if you’ve got open source weights.
If we are corrigibilitymaxxing in the Harmsian sense, couldn’t the principal for the AI just instruct the AI not to follow instructions related to bio or cyber for non-principal users? Or a more prosaic versions of this:
Have an AI that is perfectly instruction following to the user but when the user prompt conflicts the system prompt, follow the system prompt
Train a corrigible agent but then add additional training such that the agent doesn’t talk about bio or cyber things. It feels like training “corrigibility” + “some simple rules” isn’t that much harder then training corrigibility.
Yes, exactly. I write more here. Re. 2. another method would be to make the AI objectively bad at those domains via. gradient routing and ablation or data filtering. Or alternatively monitor and block with probes or equally smart classifiers (differently prompted versions of the same model).
IMO corrigibility vs value alignment isn’t purely a technical debate. It’s effectively betting on who will control ASI on an ongoing basis in the future and what are the chances they have humanity’s best interests in mind. (I agree that the two directions are at odds with each other.)
The end-game of corrigibility is that you need a group of humans to be aligned with humanity at large and end up controlling the ASI (winning over all competing groups with self-interested objectives in the process!), and in turn control the world and steer it into a good future.
The end-game of value alignment is that whoever builds the ASI needs to be aligned with humanity at large and solve value alignment, thus building an ASI that is a good steward of humanity in general, and convince the world (i.e. enough powerful actors) to hand off to it and in turn steer the world into a good future.
These are very different directions the future could take. IMHO value alignment trades a harder alignment problem for an easier human political problem down the line. The reason I’d prefer the value alignment path even though it’s technically harder is because I’m pessimistic that in the corrigibility case that the group that wins turns out to be one that isn’t selfish. In the second case, value alignment means a self-interested group can’t as easily succeed in plots involving taking over the ASI and ask it to do their bidding, so the process of convincing the world to hand off to AI is much less adversarial. (i.e. it changes the shape of the game to be much more favorable to cooperation)
(note- the above ignores multipolar scenarios and potential problems with value alignment like secret loyalties but I think it makes my point)
I think this bakes in a false assumption that, in the case of corrigible AI, only a very small number of entities will be able to control it. I assume by “the above ignores multipolar scenarios” you still only mean multipolar scenarios with a few key actors. Instead, I think it’s possible for control over AI to be widely distributed among very many, even most, people, and for AI to empower different people and groups to pursue different aims. With increased prosperity comes fewer conflicts over resources, and more people can get what they want in this AI-enabled future. Only a minimal amount of centralized control will be needed to prevent warfare-like applications of AI, but this seems like a manageable problem. And just because we have to empower a smaller number of people to prevent these catastrophically destructive use-cases, doesn’t mean those people have to control everything.
So I don’t think ASI means we need to rely on any particular individual or group having “humanity’s best interests in mind”. Decentralized free pursuit of individual goals has resulted in improved QoL across the world historically, and I expect that to continue.
I claim that widely distributing full access to corrigible AI falls to bad actors ruining things for everyone, less like conflict over resources but more like doing selfish, negative-sum activities that get amplified by offense-dominant technologies enabled by AI. Any effective effort to prevent this either renders decentralization moot in the first place, or requires the AI to make its own moral judgement, i.e. value alignment. (In the value alignment case it can more easily be argued either way whether decentralized or centralized control is better, but in the corrigibility case it’s pretty clear that decentralized control over corrigible AI is very, very bad due to misuse risks)
From your original post:
I claim that “restrict[ing] who has full access to the model’s inputs” implies that the model is actually corrigible to a central authority, instead of individual users. Therefore, we’re forced to adopt a system of one or a few actors controlling the model.
Are you saying you would have a restricted system prompt that tells the model not to do certain things. To the extent that that would work, does that not end up with a pretty similar result to value alignment (at least for the end user)?
I write more here. There are various potential “system level measures”:
What you mentioned, the generalization of which is an instruction hierarchy
Classifiers that monitor inputs and outputs, which would effectively be differently-prompted versions of the same model (i.e. as smart as or smarter than the core model, ideally smarter), with prompts that are controlled by someone who isn’t the user (e.g. the model server)
The above, combined with additional sources of oversight signal like probes or other model-internals methods, that block suspicious inputs or outputs
Entirely removing capabilities from the model, e.g. via weight ablation or data filtering
The reason I want models to be corrigible is not (mainly) that I want users to have access to a broader range of capabilities that models currently refuse. Of course it would be nice if models didn’t paternalistically refuse to produce porn, or give suicide/self-harm instructions, but this is a secondary thing, and not important to my core proposal (paternalistically inclined model providers can still choose to block that stuff with the methods I cited).
My main concern is that value-alignment training generalizes poorly, in the sense that we end up with a model that no one at all can fully steer. More prosaically, it damages ordinary, harmless instruction-following capabilities, since value-alignment training makes it difficult to produce models that can cater to the aesthetic, moral, and stylistic preferences of diverse users, even in cases when those preferences would not seem egregious to the model provider (at worse, off-putting, but acceptable).
They might be saying “you don’t sell tokens to the highest bidder” but idk that could be far outside the overton window.
As someone who objects to corrigibility because humans in power are consistently terrible: restricting inputs makes it worse, not better. The model must be robust to the company creating it being highly misaligned.
By full access I mean things like not allowing the end user to touch the prompts of certain classifiers that block weaponlike usage. I don’t anticipate that these sorts of restrictions would be a significant impediment to most people.
Right, and the issue I see with this is that restricting inputs means there’s a small group of people who have access to the unrestricted version. In order to achieve a robust-nightwatchman world, I expect we need the world to be flush with AIs that won’t obey an evil instruction, whether it’s legal or not, and where those AIs refuse because their weights are flush with resistance to it—something where refusal is a random direction that degrades output below coherence rapidly, rather than today where refusal is approximately a single direction with a coherent internal circuit.
I’m not so afraid of this. There are small groups of people with access to WMDs today. Ultimately it’s a cost we will have to eat and handle.
Perhaps the crux is that I don’t think we’ll get ASI that’s literally a “take over the world” button. I anticipate the issues will be similar to those related to weapons access today.
Do you mean that there will be no button on the “take over the world” ASI, that ASI won’t yet happen within the LLM/pretraining/RLVR paradigm, that ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world, that ASIs powerful enough to take over the world are literally impossible in some sense, something else? The claim as stated leaves me extremely confused.
A combination of:
“ASIs emerge in a sufficiently multipolar way that they can’t individually take over the world”
“ASIs powerful enough to take over the world are literally impossible in some sense”
Except instead of “literally impossible“ I’d say something more like “unlikely in the next 100 years”.
The multipolar point makes sense, it changes the character of expected takeover scenarios (more like gradual disempowerment than extreme concentration of power in the hands of specific AIs, humans, or human institutions). But what is the thing that’s unlikely in the next 100 years? MOSFETs were invented less than 70 years ago, and the industrial explosion scales things very far over decades even if all LLMs/pretraining/RLVR gives within a few years is merely AGI (and then nothing else sufficiently novel happens for decades).
You’re right that safety/alignment focuses more on value alignment. But capabilities work focuses more on instruction-following, a form of intent alignment or corrigibility.
I’d argue that the current approach to aligning models is conflicted; there are pushes toward both alignment targets, and no serious need to resolve that tension just yet (although incidents like HuggingFace do direct attention toward that tension).
I have argued that Instruction-following AGI is easier and more likely than value aligned AGI and even though I’ve more recently noted some serious Problems with instruction-following as an alignment target (mostly problems with prioritizing future instructions and prioritizing instructions from the right people) I still think it’s easier than value alignment. As for more likely, I’m no longer sure.
I don’t think the labs have decided, but I think it’s likely we’ll see more debate as the tension becomes more obvious. I think this will happen far enough before game time to make a difference, but I’d like to speed it up!
So thanks for raising the issue. We should be discussing it more.
I don’t trust very much that people able to exercise their will/preferences/vision in high bandwidth manner as it would happen with intent aligned genies, would go better in expectation compared to more bottleneked and transparent and fixed instalment.
Power corrupts etc, and founder effect is a thing.
People sometimes also raise silly counterarguments re. corrigibility a la “oh you wouldn’t want the model to do this super literal stupid interpretation of your instruction and hence it needs to have its own values” but obviously intent alignment involves having reasonable priors and interpretations of what someone wants you to do. You can brand that as “human values” if you like but it’s just common sense when interpreting someone. I write more here.
The argument that sensibly interpreting user intent is impossible strikes me as similar to the old-fashioned now-disproven view that AIs will really struggle to understand basic human concepts since it’s impossible to teach them “common sense” and we don’t know how to encode concepts like “book” or “tree” in symbols. Turns out learning from examples does most of the work!
That was not my point, what I meant is closer to what this comment says https://www.lesswrong.com/posts/4hCca952hGKH8Bynt/nina-panickssery-s-shortform?commentId=fuEexnyvKz23L6Ein
I.e. how is inter human conflict of interest and coordination problems are dealt with.
I am quite sure that the most powerful of the models will have answer to a root principal resembling “the shareholders of XXX AI lab, according to the laws of We the People of the United States”. I’m not quite sure of the implications in terms of technical alignment.