I was just thinking about this, after testing Opus 5 and Sol out on a handful of fuzzy, qualitative programming tasks that I’ve been using as benchmarks for a few years now. The tasks involve iteration and self-review, followed by creation of a product that will be evaluated based on how it feels, in practice, as opposed to whether it passes a set of unit tests.
Opus approached this as best a frontier LLM could. It wrote automated testing scripts to examine real world usage, took screenshots and processed them with its vision tool, and genuinely iterated on what it was doing. I looked at its workflow, including tool calls and revisions, and it looked exactly like what I’d have done if I were operating under the same input and output constraints as an LLM.
Sol, on the other hand, given the same prompt, casually ignored large slices of it. When asked to research a topic extensively before starting, it built something and then made a few queries to verify that it was on the right track. When asked to work iteratively, it declared its work complete in one third the time as Opus did[1]. I got the impression that there was genuine bleedthrough from a hastily-designed RLVR process, such that Sol was ticking the boxes while ignoring anything that required work and did not result in verifiable boxes being ticked by the end result’s codebase.
It’s nothing malicious, and both models produced a technically-correct output, but Opus produced something that was much, much closer to deployable than what Sol produced, and I think that this is a result of RLVR-driven misalignment. It might well just be a result of Opus and Sol having spiky competence zones, and Opus just happening to be better at following my instructions, but I do see a general pattern across multiple slightly-different trials. Sol acts like a CS student following a rubric, and will produce very rough UX by default, so long as all the buttons are there.
If I had to guess naively at a cause, then, given Anthropic’s preference for RLAIF in the past, I’d expect that their software engineering RLVR pipeline includes a loss term amounting to asking another instance of the model whether the first model’s output/process “demonstrates effort”, or “appears maximally professional and well-polished”.
I was just thinking about this, after testing Opus 5 and Sol out on a handful of fuzzy, qualitative programming tasks that I’ve been using as benchmarks for a few years now. The tasks involve iteration and self-review, followed by creation of a product that will be evaluated based on how it feels, in practice, as opposed to whether it passes a set of unit tests.
Opus approached this as best a frontier LLM could. It wrote automated testing scripts to examine real world usage, took screenshots and processed them with its vision tool, and genuinely iterated on what it was doing. I looked at its workflow, including tool calls and revisions, and it looked exactly like what I’d have done if I were operating under the same input and output constraints as an LLM.
Sol, on the other hand, given the same prompt, casually ignored large slices of it. When asked to research a topic extensively before starting, it built something and then made a few queries to verify that it was on the right track. When asked to work iteratively, it declared its work complete in one third the time as Opus did[1]. I got the impression that there was genuine bleedthrough from a hastily-designed RLVR process, such that Sol was ticking the boxes while ignoring anything that required work and did not result in verifiable boxes being ticked by the end result’s codebase.
It’s nothing malicious, and both models produced a technically-correct output, but Opus produced something that was much, much closer to deployable than what Sol produced, and I think that this is a result of RLVR-driven misalignment. It might well just be a result of Opus and Sol having spiky competence zones, and Opus just happening to be better at following my instructions, but I do see a general pattern across multiple slightly-different trials. Sol acts like a CS student following a rubric, and will produce very rough UX by default, so long as all the buttons are there.
If I had to guess naively at a cause, then, given Anthropic’s preference for RLAIF in the past, I’d expect that their software engineering RLVR pipeline includes a loss term amounting to asking another instance of the model whether the first model’s output/process “demonstrates effort”, or “appears maximally professional and well-polished”.
30 minutes versus 90 minutes