Website: pbement.com
Substack: notoneunusualthing.substack.com
DaemonicSigil
Karma: 2,122
Yeah, this one is interesting because before LLMs, there’s no expectation that the AI will be able to figure out what goal you “meant to set”. You’ll just get whatever policy maximizes the reward function (up to the limits of your RL algorithm’s ability to explore). So the mismatch between expectation and the actual learned policy basically has to exist only in the researcher’s head, because the neural net is certainly too simple to have any ideas about it. And then you can always say that the researcher (or game developers) should have expected whatever degenerate strategy the AI ended up finding.
The more recent examples are more meaningful because the LLMs have a concept of the intended space of solution strategies that they are choosing to ignore.
Datapoint: I already knew about it before reading this.
I also don’t think it’s the kind of technique that’s particularly hard to think of, once you’re trying to find ways to do the thing it does. (The thing it does is pretty different from the thing GRPO does.)
The one the sounds most surprising to have been solved to me (disclaimer: I don’t understand a lot of these) is the arithmetic circuit complexity of the permanent. Scott Aaronson suggested trying to analyze the circuit complexity of the permanent (in particular, finding the exact complexity for a small case like
More concretely, my proposal is to devote some of the world’s computing power to an all-out attempt to answer questions like the following: does computing the permanent of a 4-by-4 matrix require more arithmetic operations than computing its determinant?
I wonder if this result provides any more information about that? On the determinant side of things, there was some surprisingly recent progress here: https://arxiv.org/pdf/2301.06586
“amount that layer 52 looks at the output of layer 18” does unfortunately seem like a less interpretable quantity than attention scores between parts of the context / output, where we get to have an intuitive sense of what the tokens mean just by reading them.
Yeah, I assume they saved full thinking/output transcripts for each step and periodically saved parameter checkpoints. Depending on how often the checkpoints are, you could probably get a decent approximation of what the associated weight update was by running the trajectory through the closest checkpoint. But if you’re trying to modify the network a million steps later, probably the nearest thing to an “undo” operation is just to negatively update the already-trained network on your saved trajectory.
Then the question is how well this fixes the problem. If I reinforce misbehaviour early and then correct it at the end, does it poison the intermediate gradient steps somehow? I suppose computing the lie bracket with a few of the intermediate steps could answer the differentiable part of this. But there’s also a non-differentiable effect on token-sampling probabilities, which might be more important.
So, like, when they detected this, they didn’t keep the resulting updated weights, right? They rolled back, or maybe assigned a negative reward. Right?
This is beautiful. Thank you.
I think just training a relatively standard actor-critic / GAN might work here too? Without resorting to neuralese even. Details:
We make a GAN with a generator and discriminator network. Discriminator predicts real vs generated for outputs (CoT hidden), basically as normal. The generator gets to use chain of thought (regular non-neuralese version) to create its answer. Generator is updated by policy gradient.
GAN training wisdom suggests that it’s often good to update the discriminator more often than the generator. The sparseness of the generator training signal plays into this nicely.
But overall, this is likely not super data efficient, because we still need to update the discriminator, and the supervision there is only once per answer rather than once per token.
(Overall, I’ve become more bullish on context distillation as a very flexible and data-efficient way of doing things recently.)
I’ve been trying to use Sol to resolve an open mathematics conjecture; since now even the publicly available models are very good at this kind of thing. (See, eg the recent disproof of the Jacobian conjecture, though I began my efforts before that.)
Unfortunately, it seems that I have screwed up and picked a conjecture that happens to actually be true.
For navigation, I’d assume that wheels are easier, even for humans to control, because you don’t need to worry about balancing the robot, and humans are certainly good enough at using a joystick. Plausibly humanoid hands could be more intuitive though.
The previous strongest argument for “why make a humanoid robot?” was that the world is already designed for humans to navigate and manipulate. But there are many robot designs that could navigate the world equally well, especially if we only care about a subset of tasks and environments. The new strongest argument is that if the robot is humanoid then you can pre-train its policy on human actions.
One other area this is used I think is in finance, where people take covariances of stock movements. There’s too little data to learn the full matrix (most of the result would just be noise), so they instead learn a low-rank approximation of the covariance matrix. (Not sure the extent to which it’s common to have a continuous rank cutoff, as is done here, vs just a hard cutoff, though.)
On the other side of things, if we look at the Muon optimizer, the spectrum of singular values of a weight update tends to naturally decay smoothly, but Muon has the effect of amplifying small singular values and suppressing large ones to get a flattish spectrum. So you’d think this would amplify the effect of noise in the gradients, but I guess maybe that just ends up being helpful for neural net optimization somehow.
Link to interesting post on why Helion energy is not going to work: https://www.bhauth.com/blog/flawed ideas/helion.html
This is marked with the “Personal Blogpost” tag. You can choose hide posts with this tag from your frontpage by setting a filter.
Just wanted to note another issue with this idea. Quoting myself elsewhere:
I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all possible story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
In other words, there is a big problem with example-based training on-policy that this post ignores: If there are a huge number of answers that are all compatible with the same query, then the chain of though cannot really do much in the way of picking a particular one of them and refining it, because the supervision will be randomly selected from the set of all possible answers, so we are overwhelmingly likely to pick incorrectly. Some of the methods described above have workarounds:
The method I describe can be adjusted as follows: Sample
The MCMC sampling paper needs no modification, it already just trains on a sampled
The really smart paper does need to be modified. Probably by introducing randomness into the flow (so that the flow can model selection of one possibility from many) and then using
Thanks for the thoughts. Training a reward model is something I’d expect to require human data also. Is your picture here that training the reward model still requires human labels to train, but fewer of them (and then outputs from the reward model can be scaled easily), or that labs are directly re-purposing a standard trained model as a reward model with relatively little modification? [1]
Scraped data is cheap, and it’s not clear to me that there’s nothing one can do with it. I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all the story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
What is your first worry for alignment?
- ↩︎
Of course, even in the first case the reward model would be a fine-tune of some pretrained weights. The question is how much data is actually used to tune it.
- ↩︎
What do you make of the idea of just trying to train on a curated set of example solutions, instead of trying to create models to estimate rewards? I argued on here quite recently that this would be desirable. (And then that there were still ways to do this kind of supervision, even if we wanted to let the model think before answering.) But there are certainly drawbacks of this kind of training, for example that it is off-policy.
Talking to people in person, I’ve tried to push back on exchanging p(doom)’s on the basis of “it’s not sensible to talk about a probability that depends on one’s own actions”. And then in reply I got hit with “you really think that you personally can influence the chance that the world ends by more than a couple percent?” And I have to admit that’s a pretty good reply. I think the best one can do in response is gesture in the general direction of logical correlations between my actions and those of many other people.
Yes, obviously you train either method on everything in your training data. So there’s an implicit average, sorry for confusing you by not writing it out.
You seem to be claiming that AIME problems train more sample efficiently under policy gradient than SFT. But I assume the reward function under which such a superior result is obtained is some evaluation of whether the model solved the problem, and not
In my view, sample efficiency is very closely related to how well the model generalizes out of distribution, though they are conceptually distinct. For a very heuristic picture, you can draw a two axis graph where the horizontal axis is “# of samples the model was trained on” and the vertical axis is “how well the model generalizes” and there’s a line that goes up and to the right. And then if you train with a different architecture and you get a line that’s to the left of the first line, you say that the new architecture is more sample efficient. But if you train both architectures on your entire dataset (same number of samples), then that just looks like the model with the new architecture generalizing better.
But that’s about inductive priors, not sample efficiency.
My point is mostly sideways from those things. One thing that is very valuable about not needing a reward model is that the learner can’t learn to trick the grader. Even if your learner is strong enough to learn to trick a human grader, it doesn’t matter. Because in SFT the human just needs to produce a good answer, without ever looking at a model output. This is the main alignment advantage I am focusing on, not so much anything about OOD generalization. I could also say things here about how the need to create a grader for your problems skews the problem selection towards problems that make that easy to do. Which would make the issue of distribution shift worse.
Very nice classification! What is your take on how context distillation (or distillation in general) fits in here? To me it seems most similar to pretraining, at least mathematically. But I wonder if there are any special things that happen there that would make it worth giving its own category?
Also, do you share my intuition the pretraining/SFT category looks like the least scary one by far? Like, if you run into problems, just change the training data. Simple in principle, if not in practice due to the sheer amount of data required for pretraining.