Surely they didn’t train on games as long as Pokemon Red (or even a section thereof) with a 0-1 reward, though, so there is some kind of non-trivial generalization going on, then?
shawnghu
Karma: 309
RLVR, to wit, only works in domains with easily verifiable rewards. Capabilities elicited this way then likewise refuse to generalize to other domains
I’ve been thinking about a similar proposition recently.
What do you make of the observation that successive generations of models seemed to improve at Claude Plays Pokemon, despite not being specifically trained on it? Likewise on this robotics benchmark done by Anthropic’s frontier red team.
For context, how long are your timelines and how sure are you of them? / How sure are you that you’ll never get to use some knowledge? (Did you care very much in the past about getting to “use” the skills you learned in some direct sense?)
This point has been kind of made before https://www.lesswrong.com/posts/oKxc8maZGtnzgpNzx/rerunning-ai-safety-papers-on-every-frontier-release-would-1
Yes, this is an instance of what I’m talking about.
Suppose that you have “lower-level” and “higher-level” beliefs about yourself. For example, “I don’t eat oysters” is a lower-level belief than “I am a vegan”, which is a lower-level belief than ”I behave ethically”, which may be a lower-level belief than “I am a good person”.
People seem quite resistant to updates on their higher-level beliefs, in such as way where updates to their lower-level beliefs have a tendency to update “only locally”, i.e, in a way where their bearing on higher-level beliefs is explained away. (e.g, “I eat oysters, but they don’t have brains, so I’m effectively a vegan”, or “I’m not a vegan, but I still generally behave ethically” or “I don’t behave in ways that seem generally ethical, but I’m still a good person in some fundamental sense”.) This can be a problem inasmuch as this makes you resistant to actually updating against evidence coming at you from the outside world (such as feedback on your behaviors).
Anyway, maybe one of the epistemic functions of introspection is that it allows for higher-bandwidth “residual connections” between higher and lower-level beliefs. Especially if you have raw observational data on the history of your lower-level beliefs (including about why you did various things), you can hold these fixed as data to fit against to update your higher-level beliefs, possibly by looking at yourself as an object of inquiry and then bringing your general cognition to bear on the question.
(This also roughly goes through if you replace “beliefs about yourself” with “beliefs about your behavioral patterns”. It also goes through if instead of talking about beliefs you just talk about higher and lower-level behavioral patterns.)
Of course, the complexity of this whole setup begs the question of why it should be so hard to update your higher-order beliefs in the first place. Either there’s a good reason for that and introspection was of limited use in the ancestral environment, or human intelligence evolved in some bullshit runaway process and the human condition consists largely of countless spandrels on spandrels. I think it might be the second one.
(footnote: Here we are just assuming that “actual vegans don’t eat oysters” and that “being vegan is ethical behavior”; the facts of the matter don’t really have a bearing on my point. Perhaps I should come up with better examples.)
I actually mean this to be sort of a toy model/thought experiment of metacognition. How the hell does this actually convincingly propagate into your beliefs about X? A good theory of agent foundations should have an explanation for this.
I think this statement is obvious to people who have really focused on scaling LLMs, but recent increases in performance have substantially been about improvements to the data mixture. (This of course explains the massive investment in procuring data and RL envs.) (There’s some involvement of faster attention mechanisms, and probably some contribution from better post-training practices.)
This also helps put in perspective the late-2024 disappointment (for consumers)/lengthening of timelines that we observed after the release of GPT-4.5. One perspective on this is that returns to pretraining scaling had tapered off, because improvements on the pretraining loss didn’t help on the datasets people had (in that particular post-training paradigm).
One thing I underappreciated is the degree to which this fact has affected frontier model development. I intuitively view pretraining as this massive undertaking, and basically dismissed claims that mid-2025 LLMs like Grok 3 could possibly have spent as much compute on post-training as pretraining. (I guess I assumed this was hype or something? Dangerous heuristic to overuse.)
But in fact I think if you take Epoch’s estimates for how much compute GPT4.5 took and you use the rough “6 * num_active_params * num_training_tokens” formula, you get that Kimi-K3 probably took a roughly similar amount of compute to pretrain, and is obviously way more capable.
Also if you just take various common benchmark scores and you look at the gap between Qwen3-8B and Qwen3-32B, as well as the gap from Qwen3-32B to Qwen3.8-27B, extrapolating that a linear increase in score corresponds to an exponential increase in size, you get that approximately 2 OOMs of scale were gained by whatever people changed from Qwen3 to Qwen3.8 (data mixture and attention layers, mostly, AFAICT). (i.e, Qwen3.8-27B is as performant as a hypothetical Qwen3-2700B.) (This might be a very bad comparison inasmuch as Qwen3-8B is just distilled from 32B, and inasmuch as significantly more compute actually went into the 3.8 generation because of RL, etc.)
Here’s a fun thing for agent foundations people to ponder:
“That ‘you believe X for Y reason’ is itself a belief”. (Call this belief B.) What this implies is that when you learn that Y reason is false, you’ll update against X, and, but also if you think that Y is true and learn that B is false, you’ll update against X. This all happens at the level of your conscious awareness; often B needs to be salient to you for this update to go through. Note that this is not quite the same thing as being like a PDG modeling X and Y. (In that case, learning that Y is false automatically updates you against X.) Instead, it’s more like “believing that a PDG which models X and Y should be correct”. A strange thing to observe is that this actually convincingly propagates into your beliefs about X.
A variation of this which shares a lot of structure but seems intuitively to remove elements of introspection is that you might start by believing Y, and then you might be convinced that “Y is a reason to believe in X”. (But really, the element is still here—once convinced, you believe that Y is a reason to believe in X, as before.)
I agreed with this argument, which is why it’s particularly interesting to see several counterexamples to this conclusion in various recent events (HF incident and Hacker Opus are both meaningfully non-CDT, IIUC).
Seems that Cloud’s specific question is about why recent events have updated you, given that public information is still consistent with the hypothesis that “alignment is easy”. I think you’ve described an argument for why alignment is not easy (large amounts of optimization are scary).
I think the synthesis is something like “recent events have updated you upwards on just how scary large amounts of optimization are”? From which it follows that “no way can alignment be that easy” → “we’ve made mistakes in our thinking and what might those mistakes be”.
(Certainly I, even starting from the standpoint that alignment was not easy, was extremely surprised to see how complexly misaligned the behavior the various OpenAI swarm agents participated in was. In particular I wouldn’t have expected to see so much reward-on-the-episode gaming, modeling of the grader, inter-agent cooperation involving Schelling-type coordination and self-sacrifice, or general attempts to model the dynamics of the RL environment and task assignment algorithm. Nor would I have expected Hacker Opus’s explicit reference to the outer loop RL algorithm or reward tampering.)
It’s plausible that this has some kind of an effect, but note that there is also significant evidence that LLMs do in fact “plan” for future tokens, e.g, to avoid writing themselves into a corner when producing rhyming poetry.
https://www.anthropic.com/research/tracing-thoughts-language-model
A stronger version of this is necessary to write functional code that isn’t horrendous (way more horrendous than LLM-generated code).
(Note that via the attention mechanism, it’s possible for “thoughts” from previous tokens to be propagated to perform useful computations for the next token. It stands to reason that a very sophisticated next-token predictor would have a policy of constructing many such thoughts.)
Right, this isn’t so different from what I do in response to a lot of human writing on the internet (though the behavior is fairly strong in response to LLM-generated text).
Independent of the terrible implications for cybersec and OpenAI’s process, etc., the site seems to contain some alarming datapoints on the agents’ general situational awareness, coordination ability, and self-concept. In particular:
The agents try to crack their PRNG seed to predict what questions they’ll get in the future The agents set up heartbeats to discover when they would be terminated
These seem like a level of situational awareness that’s sufficient for a classic MIRI-style “AI in training breaking out of the box to pursue its terminal goals”.
From the linked site:
The models were running in an agentic sandbox with terminal access (and the ability to edit files within their environment) We know that the agents had access to /etc/hosts and the ability to edit this (used this to avoid the POST request restriction) We see that the agents can call curl and run setsid.
so it’s somewhat worse than even that.
It seems to me that empirical AI research/evals are automated enough that either labs or open-source could build a sort of “CI” for repeatedly checking a bunch of alignment-relevant evals upon every new model release.
If you manage to operationalize certain experiments well enough (or provide enough context for what they’re evaluating), one could imagine a pipeline where the same experiment is reproduced automatically with roughly the click of a button. (Or, to the extent that this isn’t totally automatable, yeah, you can reproducibly burn some amount of human effort doing the same experiment again and again, as long as this amount isn’t too much and you find someone willing to do this unglamorous work.)
This ranges from very small toy games such as this one or considerably more advanced/agentic evals.
You can also do the same thing to reproducibly evaluate the effectiveness of a bunch of alignment interventions that may depend on scale-dependent behavior. (e.g, maybe some old ideas didn’t work very well before, but they work well now with better models—seems like low hanging fruit to just check).
(Agreed that it’s fundamentally unavoidable, short of a bottom-up solution to alignment such as e.g ambitious mech interp or agent foundations would get us.)
I think our most recent GR paper fits your criteria, I’ll share it with you when we release. (I’m really just dragging my feet on a final set of edits and announcement post...)
I like to illustrate the setting to people by suggesting that it’s tantamount to “feeling really good and then having your memories wiped”. So later, if you happen to be in a similar context, you’ll have the vague impression that it might be good to go in this sort of a direction, for reasons ineffable to you.
(This accords with the notion of reinforcement used in, say, dog training.)
This is a fun way to illustrate how absurd it is that you can RL something into coding well.
I think other important reference points include:
this MATS 8 project: https://airtable.com/appZq2f1sM0tW9kH7/shrmkcbWdFZPBNmN6/tbljZG1LiV5rBFvh1/viwPr6ACy1zOaLSSF/recnTLwNOxwWbhCZ9
my MATS 9 project on gradient routing in RL (soon to be released),
OpenAI’s confessions work: https://openai.com/index/how-confessions-can-keep-language-models-honest/
(You can imagine recovering the scheme you devised by replacing your “something was suspect” head with a prompt to the OpenAI confessions head. You can also imagine recovering something similar to “RLVR that rewards red-teaming” by rewarding the confession head for a true confession (with/without backpropagating to the shared backbone, which has various tradeoffs).)
In all cases, there’s an important question, which is “what do you do with the signal you got which indicates that the model did a bunch of RHing”? Maybe we’re not applying direct optimization pressure against visible RHing inside the RL loop, but we’re definitely not deploying a model known to RH a bunch, so we are applying optimization pressure in the form of selection in an outer loop. It’s not even totally clear to me that this is better. Using the model’s “confessions” to patch bugs in the environment seems “asymptotically better”, but still at least suffers from the weakness that it, by design, does not catch the set of issues that models do not surface (for some reason or another). (To be clear, I think it is not obvious whether this is sufficient for alignment in the contemporary, or even AGI, case.)
yeah, but at the same time, you won’t have proved very much about alignment when the damage happens, i think. basically you’ll have predictably caused a moderate amount of damage, not very different from if you engineered a corrigible AI system to produce a moderate amount of damage.