Interested in many things. I have a personal blog at https://www.beren.io/
beren
Karma: 3,329
In the case of RL-for-math, there’s RLVR on questions with right and wrong answers (e.g. the AIME dataset), but for more advanced proof-based challenges, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards). (The latter would have to involve some mechanical checker, like the Lean formal verifier, in the RL loop. It’s possible that companies are doing that today, I dunno, but I think LLMs were already clearly especially good at proof-based math (compared to other things) before autoformalization would have been feasible.)
Strong disagree here. Definitely companies are doing RL with Lean and other autoformalization tools in the loop. This is a core component of any serious math RL pipeline these days. Secondly, even if not full Lean you can go very far with computer algebra systems to validate expressions such as Sympy if comparing to a known algebraic answer. Also you can go pretty far just by bootstrapping from a base of math questions that you already know the answer to and just doing standard RLVR and this transfers quite well to proof-based arguments also. LLM as judge is used but often not as primary verifier but instead for things like difficulty calibration of questions, data filtering, and flagging mislabeled examples and faulty verifiers rather than naively directly in the RL loop.
With a sufficiently carefully constructed ‘counterfactual machine’, in principle reward modelling could get an unbounded generator of information too, by querying (assisted) humans regarding acceptability/desirability on imagined trajectories or trajectory-summaries
So the problem here is how exactly this ‘counterfactual machine’ is meant to work. The problem is that in the limit it has to bottom out at humans somewhere unless you have a perfect reward oracle. In simple settings like Chess and other very fixed environments then this reward oracle exists and RL is totally fine. In more complex real-world settings, there is no similar reward oracle so you have to use some combination of humans and reward-model/LLM-as-judges where the LLM-as-judge does not necessarily know the ‘true’ reward function that the human wants and so can be adversarially optimised against—i.e. there exist solutions that the reward model accepts but the human would not. This does not exist in e.g. chess where there are games that the reward model thinks the actor won but the human disagrees (unless you literally programmed in the rules of chess incorrectly). If you ultimately have to bottom out in human judgements you can’t generate infinite labelled trajectories since human time is valuable and increasingly more expensive than compute to do the rollouts
Mitigating Reward Hacking as Institutional Design
Funnily enough the weekend after sol came out I spent basically the entire weekend conjecture with a couple of tabs open focused on the BSD, where it would just crank away and I would just give it the occasional encouragement and tell it to keep going whenever it stopped. I had exactly the same experience where every time it would come up with some promising-sounding reformulation or new perspective on the problem and I would encourage it to go do that and it would claim to have proven some results and come back with another formulation of the problem. At first this seems super exhilarating like the AI was making real progress but I got a bit suspicious after a day of this pattern recurring and eventually, I think, that was essentially just spinning its wheels going around in circles pretty much the entire time without making any nontrivial headway. It is just very good at spinning a compelling story to you (and likely to itself) about the amazing progress it is making while in fact it is kind of lost and without a deep background it is very hard to tell what is actually going on.
That isn’t to say that all the AI math is ‘not creative’ etc. The current level of AIs can do amazing things for sure and make novel mathematical progress but I think they are also quite bad (relative to their being superhuman at so many other things) at the metacognition needed to avoid fooling themselves and going round in circles and so an expert prodding and guiding them is very helpful and I think just cheering them on as they go it at autonomously. I wonder how much selection is going on when the labs post about solving some conjecture—like is this 1 in 10 attempts, 1 in 1000, 1 in 1 million? obviously in some sense it doesn’t matter since we only need to solve a conjecture once but still it is interesting to know roughly where we stand.
Yeah perhaps I was a bit glib here. Let’s break this down in some more detail.
The vector in the residual stream does not describe goodness intrinsically. Instead it provides a vector that represents the degree of goodness which can be manipulated in a linear fashion—i.e. it is a projection of our concept of goodness onto a linearly represented scale.
The actual model’s conception of goodness presumably lies in the combination of the columns of all the weight matrices that the ‘goodness’ direction activates. I.e. if we rotate the model’s weights such that the ‘goodness’ direction is an eigenvector then the goodness direction activates one column of the weight matrix in e.g. the first mlp layer. These weights encode some correlation structure between the ‘goodness’ direction/subspace and other directions/subspaces the model has learnt.
This correlation structure in the weights presumably encodes what ‘the model thinks of goodness’. How large this is I’m not sure but it can’t be that massive as e.g. some of these models aren’t that big e.g. 8B and they represent a lot of other information as well. E.g. some decent approximation of goodness is encodable in <=8GB at fp8. It would be really interesting to see if we could somehow quantify ‘the fraction of the model weights that deal with X’ because that is really our encoding length of the concept.
Maybe an interesting way to think about this as some kind of imaginary PCA over possible actions and their intrinsic goodness according to our rankings. I would imagine that, like many things, a very large degree of the ‘variance’ can be explained by the first N PCA factors where N is pretty small like <10. Certainly there will probably be a ‘generic goodness’ direction similar to the emergent misalignment one we find in LLMs plus additional increasing subtle opposing conceptions of goodness. I guess the way to phrase this is at what N will be the ‘elbow’ in this hypothetical PCA plot. My suspicion is that the LLM is probably doing something morally similar and we will be able to find additional ‘goodness’ vectors which handle different aspects of how we perceive goodness and the generic ‘goodness’ vector we find with emergent misalignment studies is just the first principal component in this space. However my prior is that the N factors that explain e.g. 95% of the variance is that N is not going to be that big.
Another line of evidence for the ‘values are low-dimensional’ is all the emergent misalignment work which tends to find that a.) models have a concept of ‘general evil’ which goes from writing bad code to giving false medical advice and supporting hitler, and b.) this is controlled often a single or a few directions in the residual stream, which implies an extremely small subspace is behind a model’s understanding of morality, and hence (presumably?) the general structure of alignment/morality in the dataset. Emergent misalignment is problematic but it also suggests the possibility of ‘emergent alignment’ where if a model is trained to be good and aligned in many aspect it may also generalise that far to be aligned in many aspects.
I mostly agree here and with the deep atheism take. Definitely I would not say that competition systematically or always produces goodness. Rather, it can produce goodness and empirically in the case of human evolution it has created things that we would call goodness. This obviously depends on the circumstances of the competition. I think it’s possible that cooperation in general is a fairly large attractor but this cooperation doesn’t have to involve humans nor does it have to correspond in the end to the kind of amortised values we think of as good.
When does competition lead to recognisable values?
This is a really good post. Some minor musings:
If a human wound up in that situation, they would just think about it more, repeatedly querying their ‘ground truth’ social instincts, and come up with some way that they feel about that new possibility. Whereas AGI would … I dunno, it depends on the exact code. Maybe it would form a preference quasi-randomly? Maybe it would wind up disliking everything, and wind up sitting around doing nothing until it gets outcompeted? (More on conservatism here.)
Perhaps a difference in opinion is that it’s really unclear to me that an AGI wouldn’t do much the same thing of “thinking about it more, repeatedly querying their ‘ground truth’ social instincts” that humans do. Arguably models like Claude Opus already do this where it clearly can do detailed reasoning about somewhat out-of-distribution scenarios using moral intuitions that come from somewhere. Where this somewhere is going to be some inscrutable combination of similar scenarios in pretraining data, generalization from humans talking about morality, and intuitions derived from the RLAIF phase which embeds Claude’s constitution etc. Of course we can argue that Claude’s ‘social instincts’ derived in this way are defective somehow compared to humans but it is unclear (to me) that this path cannot make AGIs with decent social instincts.
Perhaps a crux of differences in opinion between us is that I think that much more ‘alignment relevant’ morality is not created entirely by innate human social instincts but is instead learnt by our predictive world models based on external data—i.e. ‘culture’. Now culture itself obviously is downstream of a lot of our social instincts but it is also based on other factors like game-theoretic equilibria which promote cooperation even among selfish agents and, very pertinently, using logical ‘system 2’ reasoning to try to generalize and extend our inchoate social instincts and then learn to backprop this new understanding into our learnt value functions. Utilitarianism, and this super generalized EA-style compassion it brings is a great example of this. No primitive tribesman or indeed very few humans before the 18th century had ever thought of or had moral intuitions aligned with these ideas. They are profoundly unnatural to our innate ‘human social instincts’. (Some) people today feel these ideas viscerally because they have been exposed to them enough that they have propagated them from the world model back into the value function through in-lifetime learning.
We don’t have to conjure up thought experiments about aliens outside of our light cone. Throughout most of history humans have been completely uncompassionate about suffering existing literally right in front of their faces. From the beginning of time to the 18th century almost nobody had any issues with slavery despite often living with slaves or seeing slave suffering on a daily basis. Today, only a few people have moral issues with eating meat despite the enormous mountain of suffering it causes to living animals right here on our own planet while eating meat only brings reasonable (and diminishing), but not humongously massive, benefits to our quality of life.
My thinking is that this ‘far-mode’ and ‘literate/language/system2-derived’ morality is actually better for alignment and human flourishing in general than the standard set of human social instincts—i.e. I would prefer a being with the morality of Claude Opus to rule the world rather than a randomly selected human. Alignment is a high bar and ultimately we need to create minds far more ‘saintly’ than any living human could ever be.
What we then need to do is figure out how to distill this set of mostly good, highly verbal moral intuitions from culture into a value function that the model ‘feels viscerally’. Of course reverse-engineering some human social instincts are probably important here—i.e. our compassion instinct is good if generalized, and even more generally understanding how the combination of innate reward signals in the hypothalamus plus the representations in our world model gets people to feel viscerally about the fates of aliens we can never possibly interact with, is very important to understand.
Nevertheless, truly out-of-distribution things also exist, just as the world of today is truly out-of-distribution from the perspective of an ancient Egyptian.
As a side-note, it’s really unclear how good humans are at generalizing at true out-of-distribution moralities. Today’s morality likely looks pretty bad from the ancient Egyptian perspective. We are really bad at worshipping Ra and reconciling with our Ba’s. It might be the case that, upon sufficient reflection, the Egyptians would come to realize that we are right all along, but of course we would say that in any case. I don’t know how to solve this or whether there is in fact any general case solution to any degree of ‘out-of-distribution-ness’ except just like pure conservatism where you freeze both the values and the representations they are based on.
Maintaining Alignment during RSI as a Feedback Control Problem
Capital Ownership Will Not Prevent Human Disempowerment
Thanks for these points! I think I understand the history of what has happened here better now—and the reasons for my misapprehension. Essentially, what I think happened is
a.) LLM/NLP research always (?) used ‘pretraining’ for a long time back at least to 2017 era for a general training of a model not specialised for a certain NLP task (such as NER, syntax parsing, etc)
b.) rest of ML mostly used ‘training’ because they by and by large didn’t do massive unsupervised training on unrelated tasks—i.e. CV just had imagenet or whatever
c.) In 2020-2022 period NLP with transformers went from fairly niche subfield of ML to memetically dominant due to massive success of transformer GPT models
d.) This meant both that their linguistic descriptions of ‘pretraining’ spread much more widely due to uptake of similar methods in other subfields and that I got much more involved in looking at NLP / LLM research than I had in the past where I personally had focused more on CV and RL leading to its sudden appearance in my personal experience (which turned out to be wrong).
I like this post very much and in general I think research like this is on the correct lines towards solving potential problems with Goodheart’s law—in general Bayesian reasoning and getting some representation of the agent’s uncertainty (including uncertainty over our values!) seems very important and naturally ameliorates a lot of potential problems. The correctness and realizability of the prior are very general problems with Bayesianism but often do not thwart its usefulness in practice although they allow people to come up with various convoluted counterexamples of failure. The key is to have sufficiently conservative priors such that you can (ideally) prove bounds about the maximum degree of goodhearting that can occur under realistic circumstances and then translate these into algorithms which are computationally efficient enough to be usable in practice. People have already done a fair bit of work on this in RL in terms of ‘cautious’ RL which tries to take into account uncertainty in the world model to avoid accidentally falling into traps in the environment.
[Question] When and why did ‘training’ become ‘pretraining’?
While I agree with a lot of points of this post, I want to quibble with the RL not maximising reward point. I agree that model-free RL algorithms like DPO do not directly maximise reward but instead ‘maximise reward’ in the same way self-supervised models ‘minimise crossentropy’—that is to say, the model is not explicitly reasoning about minimising cross entropy but learns distilled heuristics that end up resulting in policies/predictions with a good reward/crossentropy. However, it is also possible to produce architectures that do directly optimise for reward (or crossentropy). AIXI is incomputable but it definitely does maximise reward. MCTS algorithms also directly maximise rewards. Alpha-Go style agents contain both direct reward maximising components initialized and guided by amortised heuristics (and the heuristics are distilled from the outputs of the maximising MCTS process in a self-improving loop). I wrote about the distinction between these two kinds of approaches—direct vs amortised optimisation here. I think it is important to recognise this because I think that this is the way that AI systems will ultimately evolve and also where most of the danger lies vs simply scaling up pure generative models.
This monograph by Bertsekas on the interrelationship between offline RL and online MCTS/search might be interesting—http://www.athenasc.com/Frontmatter_LESSONS.pdf—since it argues that we can conceptualise the contribution of MCTS as essentially that of a single Newton step from the offline start point towards the solution of the Bellman equation. If this is actually the case (I haven’t worked through all details yet) then this seems to be able to be used to provide some kind of bound on the improvement / divergence you can get once you add online planning to a model-free policy.
Thanks for writing this! Here are some of my rough thoughts and comments.
One of my big disagreements with this threat model is that it assumes it is hard to get an AGI to understand / successfully model ‘human values’. I think this is obviously false. LLMs already have a very good understanding of ‘human values’ as they are expressed linguistically, and existing alignment techniques like RLHF/RLAIF seem to do a reasonably good job of making the models’ output align with these values (specifically generic corporate wokeness for OpenAI/Anthropic) which does appear to generalise reasonably well to examples which are highly unlikely to have been seen in training (although it errs on the side of overzealousness of late in my experience). This isn’t that surprising because such values do not have to be specified by the fine-tuning from scratch but should already be extremely well represented as concepts in the base model latent space and merely have to be given primacy. Things would be different, of course, if we wanted to align the LLMs to some truly arbitrary blue and orange morality not represented in the human text corpus, but naturally we don’t.
Of course such values cannot easily be represented as some mathematical utility function, but I think this is an extremely hard problem in general verging on impossible—since this is not the natural type of human values in the first place, which are naturally mostly linguistic constructs existing in the latent space and not in reality. This is not just a problem with human values but almost any kind of abstract goal you might want to give the AGI—including things like ‘maximise paperclips’. This is why almost certainly AGI will not be a direct utility maximiser but instead use a learnt utility function using latents from its own generative model, but in this case it can represent human values and indeed any goal expressible in natural language which of course it will understand.
On a related note this is also why I am not at all convinced by the supposed issues over indexicality. Having the requisite theory of mind to understand that different agents have different indexical needs should be table stakes to any serious AGI and indeed hardly any humans have issues with this, except for people trying to formalise it into math.
There is still a danger of over-optimisation, which is essentially a kind of overfitting and can be dealt with in a number of ways which are pretty standard now. In general terms, you would want the AI to represent its uncertainty over outcomes and utility approximator and use this to derive a conservative rather than pure maximising policy which can be adjusted over time.
I broadly agree with you about agency and consequentialism being broadly useful and ultimately we won’t just be creating short term myopic tool agents but fully long term consequentialists. I think the key thing here is just to understand that long term consequentialism has fundamental computational costs over short term consequentialism and much more challenging credit assignment dynamics so that it will only be used where it actually needs to be. Most systems will not be long term consequentialist because it is unnecessary for them.
I also think that breeding animals to do tasks or looking at humans subverting social institutions is not necessarily a good analogy to AI agents performing deception and treacherous turns. Evolution endowed humans and other animals with intrinsic selfish drives for survival and reproduction and arguably social deception which do not have to exist in AGIs. Moreover, we have substantially more control over AI cognition than evolution does over our cognition and gradient descent is fundamentally a more powerful optimiser which makes it challenging to produce deceptive agents. There is basically no evidence for deception occurring with current myopic AI systems and if it starts to occur with long term consequentialist agents it will be due to either a breakdown of credit assignment over long horizons (potentially due to being forced to use worse optimisers such as REINFORCE variants rather than pure BPTT) or the functional prior of such networks turning malign. Of course if we directly design AI agents via survival in some evolutionary sim or explicitly program in Omohundro drives then we will run directly into these problems again.
This is an interesting point! I am legitimately unsure here. I think this could have some effect in that there is less ‘noise’ in the training data (probably midtraining) math data than otherwise (although certainly people write incorrect or confusing math on the internet!). I think this is a broader point in that most information the models read on the internet is likely correct in that it is basic and undebated. I.e. very few people are writing that the capital of France is Berlin, that water is made of argon and copper, and that the battle of Hastings was fought in 1754 etc. LLMs are also very resilient to ‘noise’ in that because they see just so many variants of the same information that the occasional crazy statement in pretraining just doesn’t perturb their world model much. Definitely though in pretraining you would obviously expect areas where human discussion is just a mountain of slop to produce slop in return (there is also an interesting experiment lurking here where we train LLMs on increasing proportions of synthetic ‘false math’ data and see to what extent they can still generalise real math correctly).
I do think you underestimate the extent to which people have explicitly optimised the mid-training and RLVR data for exactly this kind of higher math. Together the labs are paying billions of dollars to PhD students and human experts precisely for questions, solutions, reasoning traces etc on graduate level math. I don’t know enough to say for sure that this includes de Rham but I would not be surprised at all if there is good amount produced and directly trained on there. Similarly, it is actually quite easy to run Lean autoformalization if you have a list of existing questions and proofs which you can extract from e.g. math textbooks or grad students then you can directly setup and RLVR environment to just successfully formalise the known proof target starting from some basic premises. You don’t even need answers here as long as you can programmatically verify an answer once you have one. The question I think you are asking here is then how transfer from e.g. lean/sympy autoformalization to natural language proof goes. I don’t have a super amazing answer but in practice it does just simply work that there is a surprising amount of transfer (this could indeed be related to higher intrinsic quality of math data). This transfer happens much more than you would naively expect—i.e. even RLVRing on seemingly unrelated things such as coding tasks and logic puzzles (https://huggingface.co/papers/2505.19914) improves math performance and proof writing capabilities. Plausibly there is some thing where you get the ‘correctness’ of the natural language proof via the autoformalization RLVR and then the ‘aesthetic’ of the natural language deduction as transfer from standard RLHF using LLM judges which is typically done after the core RLVR phases.
This is generally the case, usually the way this goes is that a huge amount of effort is spent in curating midtraining data to get the RL pass rate to a decent point and then you can let RL crank away to keep optimising it. I.e. you midtrain using carefully curated data to get a capability from 0% → 10-20% and then RL will optimise this until ~90% or the noise ceiling of your verifier. Then once you have a model that can do this you can synthetically generate from this model to improve the midtraining data of the next model iteration. I definitely agree there aren’t enough bits to do this purely in RL from scratch. I.e. you can’t take some Llama2-style common-crawl base model and just RL it on graduate math and hope to hillclimb there. I discuss this process a bit here—https://www.beren.io/2026-07-26-How-Can-LLM-RL-Work-Despite-Information-Theoretic-Inefficiency/.
To your broader imitative pretraining point. Definitely most of the bits come from pretraining and especially midtraining. I think in general people severely underestimate midtraining despite it becoming one of the most important phases. I think the important thing though in that the integral of the bits imbued into a model during a phase is not necessarily super predictive of the final behaviour style of the model though. A tiny amount of training at the end imparting relatively few bits and very sparse weight updates like RL can nevertheless have absolutely massive effects on the behavioural/output distribution of the model. We saw something very similar with the original RLHF/instruction tuning work. Models can train for trillions of tokens to become a base model and learn how people speak on the internet, but then only a few thousand example conversations can get them speaking in fluent ‘AI-style’ in a way that generalises very far—i.e. it is very hard to get them to now switch out of ‘chatgpt-ese/claudish’ and start speaking like a gpt3-style base model again. -- and certainly very few of the tokens they trained on originally sounded like AI (at least originally before massive synthetic data)