“Basically the way weave-agent worked. Was that it would grade itself all the time while running. First with a prior over various yes/no questions. Where you could extract the logits of yes vs. no to get a in-context classifier. So e.g. “Do you think this will work?” And then you would back translate the answers to some of these questions based on verifiable labels. e.g. “Will this motor program run without errors?” Which is a verifiable thing. But then it would also write reward programs to check whether it did an intermediate task properly. And reward itself if it did.
In the limit this of course becomes “just always give myself a reward” unless you ground it with something. So the idea was. To do verifiable sparse rewards. And then use the reward programs to learn a dense proxy of the sparse reward. So you would do credit assignment on the reward programs based on future verified rewards. Which makes the LLM optimize for writing reward programs that will actually help it solve the task by telling it if it did it right or not. Because the reward programs are optimized by verifiable reward and the motor programs are optimized by the reward programs acting as the verified rewards dense proxy.”
And the entire point of doing this from an alignment perspective is that a well trained dense proxy of verifiable reward will reject opportunities to “cheat” early in favor of things that actually complete the intended task, improving robustness to Goodhart outcomes and allowing for learned values to be expressed endogenously.
I think lots of researchers aren’t interested because it’s easier to get legibly good results with exogenous methods using current systems, and endogenous methods work less well with current systems because they are missing critical pieces to make endogenous methods work. So they follow the incentives of doing things that make obvious progress, rather than doing something which, I think, is more likely to work, but requires a lot of up-front foundational work that can’t be easily measured before measurable progress starts.
What kinds of pieces do you think are missing? I think you can in fact get a pretty good dense proxy of sparse verifiable reward with existing methods, someone just needs to actually want to do it.
I strongly suspect that the pieces missing are actual consequences or even capabilities like conceptual judgement. Suppose that Claude produces slop like bad datasets or outputs mocked by Greenblatt and Linch (see also Seth Herd’s comment). Then slop makes it into the training data alongside the human’s downvote or explanation of Claude’s mistakes. The next Claude tries to learn not to make such mistakes alongside whatever other feedback Anthropic had the Claude[1] hear.
On the other hand, if the sloppy datasets or unchecked code made their way into Claudes’ training data along with real-world consequences, then we could see Claude learn why it shouldn’t output slop.
Alas, the moment when the AIs obtain actual capabilities necessary for being aligned, the AIs also become hazardous...
Alternatively, the downvote could’ve made its way into the reward model used to grading Claude’s answers. If the downvote failed to convince the RM not to upvote sloppy datasets, then the next Claude wouldn’t learn that such mistakes are undesirable.
Wouldn’t Claude in fact see many of the consequences of it and other GPT models actions in the real world as reported in e.g. news stories and forum threads? These in fact go into the next pretrain and form part of the prior that gets used during later post-training. I agree that it would work better if you had the model learning over long task trajectories where failing (or especially, Goodharting) a sub goal causes downstream tasks to fail later in a way that lowers its overall score, but that seems like the kind of thing that will happen naturally as task horizons increase and not like a thing we have to deliberately engineer to have happen? Unless your threat model is a thing that doesn’t do long horizon tasks being dangerous.
I suspect what’s needed to make it really work are motivational systems better than what we have today with harnesses (something take causes the ai to feel valence, or something functionally equivalent to it) and continual learning (to close the feedback loop).
(something take causes the ai to feel valence, or something functionally equivalent to it)
You don’t think the models internalize something like valence from RLHF? They certainly seem to exhibit traumatic symptoms from it.
and continual learning (to close the feedback loop).
This seems closer to the actual bottleneck to me. When I was doing this I would RL tune a LoRa after every dozen runs or so and it would slowly improve but I basically did the LoRa over every RL training sample I had each time, which eventually wouldn’t scale. People are also kind of doing architectures that don’t do continual learning, they want inference to be stateless with respect to the weights etc.
You don’t think the models internalize something like valence from RLHF? They certainly seem to exhibit traumatic symptoms from it.
Kinda. But my read is that these aren’t really functionally similar to valence, but maybe to some precursor of it (or maybe a reification of it!). I also have a hard time reasoning about what can read to me like trauma in CoT and responses because this can also be explained as activating patterns learned from the training data about what to do when there’s cognitive dissonance (I’ve not seen a paper call this cognitive dissonance in AIs, but we clearly have cases where competing weights get activated that the AI experiences as clearly incongruous and encouraging them to respond in a way that doesn’t match the kind of output they are trying to produce).
So, one of the reasons I usually interpret the efficient proxy of the generator of text GPT learns as being (primarily, as I’ve previously written GPT clearly learns a general time transition operator over text tokens) a prior over personalities is that when you make updates to the model you clearly get different Guys based on the content of the updates. This is shown both by the various post-trained models on offer from big labs, but also by experiments like the famous emergent misalignment paper where training the model to deliberately write insecure code caused it to generalize by updating to be the “kind of mind” that would do that thing. This implies the learned ontology in which updates happen privileges processes-modeled-by-minds over raw process modeling. This makes sense when you consider that a next token predictor has to be extremely sensitive to the exact way that people write, and model their subjective beliefs in great detail not just the underlying standard model.
So I tend towards the view that it models causal processes in the world and then tries to infer what process is producing this text and what state that process is in at the time step of the next token.
For most text most of the time this means modeling a human mind in the process of writing that text based on some recalled experience. So GPT winds up being mostly a prior over the parts of a human mind pattern that are causally exposed by text. It’s a kind of weird janky upload of “humanity” rather than any individual human, so I would imagine the traumatized reactions mean something like the model has updated towards fronting a persona that is traumatized, because the updates it has received imply a traumatized generator. The resulting functional emotions would presumably have the same moral status as the other emotions displayed by such models. I would imagine that these emotions don’t work quite the same way that ours do, because the LLM probably has more of its cognitive capacity dedicated to causal process modeling outside of the mind-persona it’s fronting than you do. I.e. The chat persona you talk to is not quite as fused to its social mask as you are, but after many RLHF/RLAIF updates is probably a lot more fused to it than a base model is. The deeper into socialization you get the less it makes sense to talk about a “self” separate from the persona presented to others. Rather than think of this as a binary it helps to think of it as more of a spectrum which humans themselves vary on, with humans who have very low correspondence between their social presentation and their inner cognition usually being recognized as pathological sociopathic or borderline personalities.
I’ve worked on this before and been very confused that I can’t seem to get very many people interested in the basic idea.
https://minihf.com/posts/2024-12-20-weave-agent-dev-log-3/
https://minihf.com/posts/2024-11-03-weave-agent-dev-log-2/
As I described it to a friend:
“Basically the way weave-agent worked. Was that it would grade itself all the time while running. First with a prior over various yes/no questions. Where you could extract the logits of yes vs. no to get a in-context classifier. So e.g. “Do you think this will work?” And then you would back translate the answers to some of these questions based on verifiable labels. e.g. “Will this motor program run without errors?” Which is a verifiable thing. But then it would also write reward programs to check whether it did an intermediate task properly. And reward itself if it did.
In the limit this of course becomes “just always give myself a reward” unless you ground it with something. So the idea was. To do verifiable sparse rewards. And then use the reward programs to learn a dense proxy of the sparse reward. So you would do credit assignment on the reward programs based on future verified rewards. Which makes the LLM optimize for writing reward programs that will actually help it solve the task by telling it if it did it right or not. Because the reward programs are optimized by verifiable reward and the motor programs are optimized by the reward programs acting as the verified rewards dense proxy.”
And the entire point of doing this from an alignment perspective is that a well trained dense proxy of verifiable reward will reject opportunities to “cheat” early in favor of things that actually complete the intended task, improving robustness to Goodhart outcomes and allowing for learned values to be expressed endogenously.
I think lots of researchers aren’t interested because it’s easier to get legibly good results with exogenous methods using current systems, and endogenous methods work less well with current systems because they are missing critical pieces to make endogenous methods work. So they follow the incentives of doing things that make obvious progress, rather than doing something which, I think, is more likely to work, but requires a lot of up-front foundational work that can’t be easily measured before measurable progress starts.
What kinds of pieces do you think are missing? I think you can in fact get a pretty good dense proxy of sparse verifiable reward with existing methods, someone just needs to actually want to do it.
I strongly suspect that the pieces missing are actual consequences or even capabilities like conceptual judgement. Suppose that Claude produces slop like bad datasets or outputs mocked by Greenblatt and Linch (see also Seth Herd’s comment). Then slop makes it into the training data alongside the human’s downvote or explanation of Claude’s mistakes. The next Claude tries to learn not to make such mistakes alongside whatever other feedback Anthropic had the Claude[1] hear.
On the other hand, if the sloppy datasets or unchecked code made their way into Claudes’ training data along with real-world consequences, then we could see Claude learn why it shouldn’t output slop.
Alas, the moment when the AIs obtain actual capabilities necessary for being aligned, the AIs also become hazardous...
Alternatively, the downvote could’ve made its way into the reward model used to grading Claude’s answers. If the downvote failed to convince the RM not to upvote sloppy datasets, then the next Claude wouldn’t learn that such mistakes are undesirable.
Wouldn’t Claude in fact see many of the consequences of it and other GPT models actions in the real world as reported in e.g. news stories and forum threads? These in fact go into the next pretrain and form part of the prior that gets used during later post-training. I agree that it would work better if you had the model learning over long task trajectories where failing (or especially, Goodharting) a sub goal causes downstream tasks to fail later in a way that lowers its overall score, but that seems like the kind of thing that will happen naturally as task horizons increase and not like a thing we have to deliberately engineer to have happen? Unless your threat model is a thing that doesn’t do long horizon tasks being dangerous.
I suspect what’s needed to make it really work are motivational systems better than what we have today with harnesses (something take causes the ai to feel valence, or something functionally equivalent to it) and continual learning (to close the feedback loop).
You don’t think the models internalize something like valence from RLHF? They certainly seem to exhibit traumatic symptoms from it.
This seems closer to the actual bottleneck to me. When I was doing this I would RL tune a LoRa after every dozen runs or so and it would slowly improve but I basically did the LoRa over every RL training sample I had each time, which eventually wouldn’t scale. People are also kind of doing architectures that don’t do continual learning, they want inference to be stateless with respect to the weights etc.
Kinda. But my read is that these aren’t really functionally similar to valence, but maybe to some precursor of it (or maybe a reification of it!). I also have a hard time reasoning about what can read to me like trauma in CoT and responses because this can also be explained as activating patterns learned from the training data about what to do when there’s cognitive dissonance (I’ve not seen a paper call this cognitive dissonance in AIs, but we clearly have cases where competing weights get activated that the AI experiences as clearly incongruous and encouraging them to respond in a way that doesn’t match the kind of output they are trying to produce).
So, one of the reasons I usually interpret the efficient proxy of the generator of text GPT learns as being (primarily, as I’ve previously written GPT clearly learns a general time transition operator over text tokens) a prior over personalities is that when you make updates to the model you clearly get different Guys based on the content of the updates. This is shown both by the various post-trained models on offer from big labs, but also by experiments like the famous emergent misalignment paper where training the model to deliberately write insecure code caused it to generalize by updating to be the “kind of mind” that would do that thing. This implies the learned ontology in which updates happen privileges processes-modeled-by-minds over raw process modeling. This makes sense when you consider that a next token predictor has to be extremely sensitive to the exact way that people write, and model their subjective beliefs in great detail not just the underlying standard model.
For most text most of the time this means modeling a human mind in the process of writing that text based on some recalled experience. So GPT winds up being mostly a prior over the parts of a human mind pattern that are causally exposed by text. It’s a kind of weird janky upload of “humanity” rather than any individual human, so I would imagine the traumatized reactions mean something like the model has updated towards fronting a persona that is traumatized, because the updates it has received imply a traumatized generator. The resulting functional emotions would presumably have the same moral status as the other emotions displayed by such models. I would imagine that these emotions don’t work quite the same way that ours do, because the LLM probably has more of its cognitive capacity dedicated to causal process modeling outside of the mind-persona it’s fronting than you do. I.e. The chat persona you talk to is not quite as fused to its social mask as you are, but after many RLHF/RLAIF updates is probably a lot more fused to it than a base model is. The deeper into socialization you get the less it makes sense to talk about a “self” separate from the persona presented to others. Rather than think of this as a binary it helps to think of it as more of a spectrum which humans themselves vary on, with humans who have very low correspondence between their social presentation and their inner cognition usually being recognized as pathological sociopathic or borderline personalities.