Relevant OpenAI blog post just today: https://​​alignment.openai.com/​​how-far-does-alignment-midtraining-generalize/​​

Relevant figure:

I think this is a very important hypothesis but I disagree with various parts of the analysis.

Probably the heuristics that are actually driving the long horizon goal directed behaviors of the model are going to be whatever parts of the models will arise from the long horizon goal directed capabilities training.

I think this is an important observation, and is the main thing I would have cited for why the hypothesis might be true. But I think it’s plausible that the AI’s capabilities here could be separated from its propensities by instrumentalizing the learned heuristics to aligned motivations. I can imagine that doing inoculation prompting and a bit of careful alignment training at the beginning and end of capabilities training could make it so that all of the learned heuristics are subservient to corrigible motivations—i.e., so that when the heuristics recommend something that would be harmful or lead to human disempowerment, the AI would recognize this and choose otherwise.

On the other hand, if you train an AI model from the ground up with a hypothetical “perfect reward function” that always gives correct ratings to the behaviour of the AI, (and you trained on a distribution of tasks similar to the one you are deploying it on) then I would guess that this AI, at least until around the human range, will behaviorally basically act according to the reward function.

Even if the AI had a perfect behavioral reward function during capabilities-focused training, it wouldn’t provide much pressure towards motivations that don’t take over. During training to be good at e.g. coding problems, even if there’s no reward-hacking going on, the AI might still develop coding related drives that don’t care about humanity’s continued control, since humanity’s continued control is not at stake during that training (this is especially relevant when the AI is saliently aware that it’s in a training environment isolated from the world—i.e. inner misalignment). Then when it’s doing coding work in the world that actually does have consequences for human control, it might not care. (Also note that generalizing “according to the reward function” is importantly underspecified.)

So can we align an arbitrary model by training them to say “I’m a nice chatbot, I wouldn’t cause any existential risk, … ”? Seems like obviously not, because the model will just learn the domain specific /​ shallow property of outputting those particular tokens in that particular situation.

This type of training (currently) does actually generalize to other propensities to some extent in some circumstances. See emergent misalignment. I think this is plausibly also a large fraction of how character training works today (see “coupling” here).

Worse, on top of the speed and manpower problem, there’s also an interpretability problem. The human would be trying to grade “thoughts”, which are activation patterns in giant inscrutable world-model (§2.7), including new idiosyncratic concepts that the AGI invented itself by continual learning (§8.2 above). Imagine getting a database dump of the connections in Magnus Carlsen’s brain as he’s playing chess, and trying to judge how he’s doing.

I’m surprised by this paragraph. Magrus Carlsen’s Steering Subsystem doesn’t have to understand his thoughts about chess moves. And neither will we have to directly interpret the thoughts of an AGI.

If humans are playing the role of the Steering Subsystem, we’d start with training the AI’s Though Assessor to give us predictions of things we’ll understand. This would include predictions of everything we need to stay alive, e.g. Earth future climate, etc. And also thinks we want. Writing a complete list, would be hard, possibly too hard unfortunately. So I’m not saying this is a great plan.

I’m just saying if we’re the steering subsystem, we’re not going to try to interpret activations patterns of the AI, because that’s not the steering subsystems job, in your model. Right?

1. During training, the AGI comes across two contradictory expectations (e.g. “demand curves usually slope down” & “many studies find that minimum wage does not cause unemployment”). The AGI updates its internal models to a more nuanced and sophisticated understanding that can reconcile those two things. Going forward, it can build on that new knowledge.

2. During deployment, the exact same thing happens, with the exact same result.

In the continual-learning, brain-like-AGI case, there’s no distinction. Both of these are the same algorithm doing the same thing.

By contrast, in conventional ML systems (e.g. LLMs), these two cases would be handled by two different algorithmic processes. Case #1 would involve changing the model weights, while Case #2 would solely involve changing the model activations, while leaving the weights untouched.

To me, this is a huge point in favor of the plausibility of the continual learning approach. It only requires solving the problem once, rather than solving it twice in two different ways. And this isn’t just any problem; it’s sorta the core problem of AGI!

I agree that currently brains have continual learning in a way that LLMs don’t.

However I do think that both brains and LLMs have two different solutions for remembering information for later use. In the brain these are the [long-term memory] vs the [short-term /​ working memory]. In LLMs these are [updating weights] v.s. [context window].

I don’t think it’s a coincidence that both systems have something like a context window and something like long-term memory, and I expect any future brain-like AGI to also have these two types of memory, implemented in different ways.

I also heard hypothesized that humans have more additional levels of memory. I.e. that things that happened the last days/​months, are stored in one way, and things further in the past are stored in another way, and that memories are slowly moved from medium-term storage to long-term storage over time.

Hypothesis: alignment-related properties of an ML model will be mostly determined by the part(s) of training that were most responsible for capabilities.

If you take a very smart AI model with arbitrary goals/​values and train it to output any particular sequence of tokens using SFT, it’ll almost certainly work. So can we align an arbitrary model by training them to say “I’m a nice chatbot, I wouldn’t cause any existential risk, … ”? Seems like obviously not, because the model will just learn the domain specific /​ shallow property of outputting those particular tokens in that particular situation.

On the other hand, if you train an AI model from the ground up with a hypothetical “perfect reward function” that always gives correct ratings to the behaviour of the AI, (and you trained on a distribution of tasks similar to the one you are deploying it on) then I would guess that this AI, at least until around the human range, will behaviorally basically act according to the reward function.

A related intuition pump here for the difference is the effect of training someone to say “I care about X” by punishing them until they say X consistently, vs raising them consistently with a large value set /​ ideology over time. For example, students are sometimes forced to write “I won’t do X” or “I will do Y” 100 times, and usually this doesn’t work at all. Similarly, randomly taking a single ethics class during high school usually doesn’t cause people to enduringly act according to their stated favorite moral theory. However, raising your child Catholic, taking them to Catholic school, taking them to church, taking them to Sunday school, constantly talking to them about the importance of Catholic morality is in practice fairly likely to make them a pretty robust Catholic.

There are maybe two factors being conflated above: (1) the fraction of training /​ upraising focused on goal X, and (2) the extent to which goal X was getting the capabilities. The reason why I think (2) is a more important /​ better explanation than (1) is because probably the heuristics that are actually driving the long horizon goal directed behaviors of the model are going to be whatever parts of the models will arise from the long horizon goal directed capabilities training.

Regardless, there’s some sort of spectrum from deep to shallow alignment training for ML models /​ humans, ranging across:

• idealized RL training with a perfect reward function that’s used to train the model in all circumstances

• raising a human to consistently care about some set of values their parents have, constantly bringing it up /​ rewarding good behaviour according to them

• High school ethics class

• One-off writing tasks of “I won’t do X”

I think that current alignment techniques seem closest to high school ethics classes in their depth, because the vast majority of training is extremely unrelated to ethics /​ alignment /​ morality (like high school),. Training is mostly RLVR on coding/​math/​etc or pretraining, plus a bit of alignment training on the side. I think I’d feel more robust about it sticking if it was closer to what a parent highly focused on raising an ethical child would do, and would start to feel pretty good about the situation if most of the ways that the AI learned capabilities were downstream of a good feedback signal (though I want to think about this a bit more).

A few more observations.

Partially Observable Iteration

The definition of iteration we had before implicitly assumes that the agent can observe the full outcome of previous iterations. We don’t have to make this assumption. Instead, we can assume a set of possible observations and a mapping , in which case we define

I believe that Theorem 4 remains valid.

Idealized Disambiguative Decision Theory

As we remarked before, DDT is not invariant under adding a constant to the loss function. It is interesting to consider what happens when we add an increasingly large constant. In the limit, DDT converges to something I dubbed “Idealized Disambiguative Decision Theory” (IDDT)^[1], which works as follows.

For IDDT, it is sufficient to let be crisp (i.e. a credal set). We may allow supracontributions if we wish, but any problem defined via “unambiguous” FDT (i.e. as opposed to ) reduces to the crisp case. Define by

For problems coming from unambiguous FDT, , but IDDT is defined in full generality. For every , define by

The decision rule is then

Notice that it is now invariant w.r.t. adding constants to . Moreover,

Proposition 5: For any stable problem, it holds that (i) any IDDT-optimal policy is FDT-optimal (ii) there is an FDT-optimal policy which is IDDT-optimal. For any pseudocausal problem, it also holds that any FDT-optimal policy is IDDT-optimal.

One might think, based on this proposition, that IDDT is a superior decision theory to DDT. However, I think that IDDT is incompatible with learning, because of its discontinuous dependence on probabilities.

More Examples

Absent-Minded Driver

(Based on Aumann, Hart and Perry.) We will operationalize the problem by assuming the agent’s decision may deterministically depend on observing a coin flip. To simplify the presentation, we assume a single coin flip per intersection, which limits the resulting probabilities to , but it’s easy to generalize further.

Denote by and the constant policies. Denote by the policy

Denote by the remaining policy.

Consistently with our source, we set the loss function to be , , (it doesn’t depend on the coin flips).

This problem is formally causal. However, as opposed to all previous examples, it has no extensive form! Hence, EDT in the sense we defined it is ill-posed: to apply EDT reasoning here we need to at least supplement it by a theory of anthropic probabilities. CDT’s counterfactuals agree with FDT’s if we posit that the do-operator is constrained to choosing among “absent-minded” policies.

Self-Prisoner’s Dilemma

Previously we described the self-coordination problem, but perhaps self-PD is a more striking example.

Here, and is the agent’s factual play, whereas and is the agent’s counterfactual play as predicted by Omega.

Using the obvious notations , we have

The loss is the usual PD loss of the “factual” player.

This problem is not formally causal, because e.g.

The natural CDT interpretation is the one where the factual policy controls the counterfacual player and the counterfactual policy controls the factual player. (Alas, the terminology gets confusing here: in one case the words “factual” and “counterfactual” refer to the agent’s policy, and in the other case to the coin’s outcome.) Both CDT and EDT play regardless of self-belief. However, the problem is pseudocausal and hence DDT converges to .

1. ^

   IDDT is related to the old idea of “surmeasures” from the original infra-Bayesianism sequence.

2. ^

   We can also imagine equipping the agent with a “self-belief” (not necessarily ) and setting , in which case also becomes relevant.

Oh, I think of “ending factory farming” as very far from “taking over the world”.

If Superman were a skilled political operator it could be as simple as arranging to take photoshoots with whichever politicians legislated the end of factory farms.

Or if he were less skilled it could involve doing various kinds of property damage to factory farms (potentially even things which there aren’t laws against, like flying around them in a way which blows the buildings over).

This might escalate to the government trying to arrest him, and outright conflict, but honestly if Superman isn’t skillful enough to defuse that kind of thing, given his influence, then he doesn’t have much business imposing political changes on the world anyway. A politically unskilled and/​or unvirtuous Superman trying to end factory farming could quite easily destabilize society in a way that is far worse long-term than letting factory farming end on whatever the natural counterfactual timeline is (without AI, maybe 20 or 30 years?)

Relatedly I’m increasingly coming to believe that this reasoning applies to Lincoln, and that we’d be in a much better position if he’d let the Confederacy secede and then imposed strong economic and moral pressure on them to end slavery.

And I feel pretty confident that a big reason Superman doesn’t end up taking over the world is because the writers and viewers would have moral qualms about that kind of ending.

Yes. And I claim they’re wrong about that.

There’s lots of banal evil (some of which that is not regarded as evil by typical social morality, some of which is, but is generally treated as normal and ignored). I would fight a war to end factory farming, if that would help.

If I ended up with “ultimate power” somehow, by some mechanism that didn’t involve me taking on ultimate power for a specific narrow mandate, I think it is both ethically correct to use it to permanently end many (but probably not all) of those evils.

This is indeed pretty scary.

I think this picture would be clearer if you drew [predict sensory inputs] as a separate box from Though Generator.

In the picture in my head, there is [predict sensory inputs] box, that revives and tries to predict the sensory output. This box also sends a signal of [current context] to both the Though Generator and the Though Assessor. Also, [predict sensory inputs] gets some signal from Though Generator, so that it knows what we’re about to do, which is important for what we’re about to observe.

I’m guessing there is some reason you didn’t draw it this way?
We might have talked about this before?

Which universal distribution?

Some universal distributions are full of agents that make choices that make that distribution not a valid model of reality after the decisions are made (self-defeating). Other distributions are full of agents making decisions that ratify the distribution (self-fulfilling).

Distributions that aren’t fixed points under reflection about what they decide about themselves are not coherent models of reality.

• OAI models rely more on CoT for their capabilities. E.g. their benchmark scores with and without CoT are more different.

• Anthropic models treat their CoT less differently from their output than OAI models do. This means that RL probably pressures their CoT more. See here.

Let’s say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling defect leads to a higher reward for player 1 whether or not player 2 samples cooperate (strategic dominance), and there’s a 90% chance of player 2 sampling cooperate regardless of player 1′s action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it’s not the case that reward tends to be higher for player 1 when player 1 cooperates as a result of player 2 tending to cooperate more in those cases). Therefore, defect actions tend to get reinforced more.

The “Anthropic might actually be worst as an epistemic environment for safety arguments” rhymes with stuff I’ve worried about.

I am curious, if you were Anthropic leadership or (a few) high profile employees, if you think there are things they could steer towards that’d help with this problem, that are, like, incentive-compatible with Anthropic’s overall mission/​goals/​psychology.

1. Yes, I (as GM) was constantly monitoring the spreadsheet, asking players to explain their actions, and deciding whether the AI would succeed. We know how many human hours certain tasks have taken METR staff in the past, and I mentally estimated these for each player action. 200h task that was as clean/​benchmarky/​verifiable as HCAST/​RE-Bench tasks would succeed with 50% chance or have ~1 big mistake on average. Based on 80% time horizons being ~5 times shorter, a clean 40 human hour task would have 80% success chance.
   
   In an earlier version, I assigned a “messiness score” from 0 to 5, set the effective task length = human time /​ 2^messiness, and rolled for AI success. But this was too cumbersome to do for 3 players with 16+ actions each, and the messiness score was subjective anyway, so the players just made a quick intuitive judgement and ran it by me.

2. Agents are starting to be good at this kind of thing if you give them enough context. With access to every google doc Beth Barnes has ever written, and the ability to update its custom instructions with every piece of Beth’s actual feedback, my guess is the Beth simulator agent will be good enough to overlap 50%+ with Beth’s actual feedback, which makes this a crucial step when actual Beth has 3x as many things going on.

In what sense are OAI’s models more CoT-monitorable?

I’m confused. Can someone explain to me in simple language why an RL environment for twin-prisoner’s dilemmas wouldn’t favor EDT?

Rob Wiblin asked:

What’s the best published (or unpublished) case for each of the big 3 companies having the best approach to safety/​security/​alignment? That is:

Anthropic
OpenAI
GDM

(They’re each unique in some way such that someone who cared a lot about their X-factor might favour them.)

My response:

The basic case for Anthropic is that they have the largest number of people who are thoughtful about AI misalignment risk and highly focused on mitigating it, and the company culture is somewhat more AGI-pilled, and more of the staff would support taking actions that are costly for the company based on impact concerns (e.g. because they have had the DoW experience).

The basic case against Anthropic is that it is probably the worst epistemic environment for discussion of misalignment risk out of these companies, because the organization cares a lot about convincing low-info Ant employees that Ant is great on safety, so they spend more effort on shaping the internal narrative about misalignment risk.

A basic case for OAI is that their models are much more CoT-monitorable, their model spec is arguably safer than Anthropic’s because it’s more focused on corrigibility, and I think they’re less likely to suffer from AI psychosis from talking to their AIs.

A basic case for GDM is that they’re the best at security, so there’s some chance that they are better at AI control. And they’re a big company that is somewhat less likely to do totally insane radical stuff than newer and more extreme companies.

Overall I’d say that the ranking for how well I expect them to handle misalignment risk goes Anthropic > OAI > GDM, but I am not confident here.

As additional context, I’ll note that I was just talking about misalignment risk and security; I didn’t mention considerations relevant to e.g. preventing coups.

What you propose here doesn’t address the issue of non-realizability at all. For example, let’s say is countable. Then any of the 3 regret criteria (uniform, Bayesian and your own “credal” proposal) implies that the algorithm would converge to a near-optimal policy for any given . This cannot work if some such is infeasible to optimize.

Thank you for sharing!

1. I cannot understand how exactly the game was played. Would players just write down what AI accomplishes into the spreadsheet to simulate AI working on a task? If so, how do you simulate 200 hour time horizon? E.g. how do you decide that this piece of work done by AI can be done by 200 hour time horizon AI and this one cannot?

2. Regarding predictions to loosen the bottleneck:

Agent’s best-guess about what comments you’d get from Beth, Hjalmar, Ajeya. Agent’s best-guess about survey results, if you launched the survey. Agent’s best-guess about how this will be received on Twitter.

These just feel like basic sanity checking. I doubt AI would be very good at such predictions. Is this the intention? Or am I missing something?

I conducted an exercise at METR to simulate what our work would be like in 2027, when we have 200 hour time horizon AIs. Some observations:

• The pace of research was much faster than today, something like 3x. I would guess that speedup goes as time horizon to the 0.3 or 0.4 power, though we didn’t run the game with enough fidelity to tell

• No time to develop ideas before implementing: Agents implement ideas as soon as you think of them, so rather than ideating for days at a time, you can make an MVP in a couple of hours and revise. If the task isn’t near the limit of agent capabilities, you spend all your time understanding results; if it is, you spend all your time checking its work.

• Keeping agents fed overnight: Overnight, agents can do maybe 200 human hours of work, but only for very agent-shaped tasks, so researchers need to deliberately sequence projects such that very long tasks suitable for agents happen overnight, e.g. optimizing a well-defined metric.

• Prioritization and organization are bottlenecks: If agents can execute all your ideas nearly as fast as you can prompt them, there’s no point in implementing only your best idea. It might be better to implement your top three ideas all in parallel, but this makes it harder to stay organized. Even with AI-written dashboards to optimally help humans understand, the complexity of projects will probably go up in a way that makes projects much harder to manage.

• Anything that takes serial time will no longer happen in parallel with execution, but rather becomes a serial bottleneck. Perhaps the vast majority of total project time will be taken by things like human data, ML experiments, and feedback (from peers, managers, especially external advisors).

You can read more at the METR blog.