Scalable oversight at Resolution
Jacob Pfau
Karma: 1,771
Models don’t seem to be dishonest in the way humans are
Debate with Self-Play Best-of-N Optimization
Announcing our $160M grant from Coefficient Giving
Research update: RL on Debate Games shows Proposal Accuracy uplift alongside Judge Hacking
From asking Claude, my understanding is that such an EMP would not itself wipe hard drives. So it wouldn’t directly, irreversibly “kill” an AGI. Still seems like a relevant move to consider!
Resolution (fka Sequent): scale and automation for higher confidence in alignment
Who is the median AI safety researcher? Probably they are new to AI safety, and maybe they are new to research. In which case, I do not think they should be much more ambitious. If you mean the ‘experience-weighted median researcher hour’ or something similar then I’d likely agree.
A bit of a nitpick, but there’s high value to narrowly-scoped research for getting tight feedback loops. On the other hand, favoring such problems can skew the community view of what is important and what research is in general. I see that as more of a comms problem than anything though.
Automated Alignment is Harder Than You Think
Claim: The importance of interface design rises proportionally to the degree of non-conscientiousness of the collaborator. AIs are very smart but very non-conscientious therefore interface design matters a great deal.
Completely agree. I was sloppy and quoted the wrong thing. I’ve edited in to fix: meant to say ‘”remove all human coders” is ambiguous.’ rather than the old text.
From AI Futures
An [automated coder] AC, if dropped into present day, would be as productive on its own as only human coders with no AIs. That is, you could remove all human coders from the AGI project and it would go as fast as if there were only human coders. The project can use 5% of their compute supply to run the AC.
What are people’s probabilities that the AC bar has already been reached?
And what is the right operationalization? “remove all human coders” is ambiguous. To make this precise, I have in mind removing all staff who are best described as individual contributors to the lab’s engineering.
From personas to intentions: towards a science of motivations for AI models
This is fiction which uses excerpts from Palantir CEO Karp’s actual talk. Here’s an actual clip
Here’s Claude’s compilation of actual quotes (I have not verified them). a16z American Dynamism Summit 2026 · March 3, 2026
“If Silicon Valley believes we are going to take away everyone’s white-collar job … and you’re gonna screw the military, if you don’t think that’s gonna lead to nationalization of our technology, you’re retarded.”
“You might be particularly retarded, because you have a 160 IQ.”
“[They are] primarily Democratic shaped people that you and I grew up with, highly educated people who went to elite schools or went to schools that are almost elite for one party.”
(referring to white-collar workers at risk of AI displacement)
“There is a lot of subtlety here behind the curtain, and I’ve been heavily involved in that subtlety. Where [AI] can be deployed, what can be deployed — there is a difference between the US military and surveillance.”
“Despite what everyone thinks, Palantir is the anti-surveillance company.”
“The danger for our industry is that you get a famous horseshoe effect, where there is only one thing people agree on, and that’s that this is not paying the bills and our industry should be nationalized.”
Sources: Gizmodo, Benzinga, Stocktwits/Asianet, Sherwood News, The Information — all citing video posted to X by Katherine Boyle (a16z) and reporter Maya Sulkin (The Free Press). No full transcript of the session has been published as of March 4, 2026.
Seems sub-optimal that none of them have backgrounds in AI.
Thanks for flagging this post from Jan! I wrote up a response to it that I’m copying over here:
I think there’s a structural risk in the transition to automated alignment research that your post doesn’t address, and is easy to miss when focusing on the capabilities question of whether models can do fuzzy tasks.
I claim automated alignment faces correlated fuzzy errors that are (a) produced by correlated models with no independent error correction to catch them, (b) embedded throughout the research workflow so you can’t isolate them, (c) increasingly shaped by unreliable supervision so they’re becoming more likely. I think this is a more serious obstacle to automated alignment research than the capabilities question of whether models can pass a research taste bar on the average case.
(a) Correlated error: Human science is trustworthy not because individual researchers have great fuzzy judgment, but because independent errors get corrected by institutional and individual diversity. Different groups work independently, make different mistakes, and cross-critique each other — scientific progress self-corrects errors. Automated alignment research with one model (or a few closely related models) introduces a field-wide correlation breaking the error-correction mechanism. So the fuzzy-task problem is more than “can the model do it well enough”, it’s “can we detect correlated subtle failures in judgment when there’s no independent cross-check?”
(b) Non-modularity of fuzzy judgments: This gets worse when you notice that fuzzy judgment isn’t a separably auditable module, it’s threaded through all parts of the automated research process. When a model writes a new codebase, it makes dozens of fuzzy calls: how to structure the code for legibility, what to plot, what patterns look interesting enough to flag. Every long-horizon “crisp” task is filled with this kind of judgment. Your proposed mitigations implicitly assume you can identify where the fuzzy judgment lives and check it. But if it’s pervasive, you can’t afford to double-check all of it. The attack surface for correlated subtle errors is the entire research workflow, not a bounded subset of “fuzzy tasks.”
(c) Increasing fuzzy error rate: I worry that generalisation to good judgments is getting less likely. Right now the mapping between verifiable work and fuzzier judgment (e.g., between a proof and a natural-language explanation of that proof) is reasonably well-anchored by human data—humans write up their proofs for communication—not just lean! We know what good code structure looks like; we know what a clear plot looks like. As models operate past human capability frontiers, that anchoring weakens and preference data plays an increasingly large role in shaping fuzzy generalization — a much shakier training signal.
Examining intuitions around discontinuity driven by recursive self improvement (RSI)
I had a couple un-examined intuitions that made the case for abrupt takeoff triggered by self-aware RSI appear plausible in my mind. I’ll lay out a couple lines of intuitions regarding why RSI might lead to discontinuity in capabilities and then debunk them. On reflection I believe that rich forms of self-awareness in RSI as entirely compatible with gradual takeoff. There are other, possibly better, intuitions for RSI takeoff though; for instance my below points do not address super-exponential progress from automated researchers!
My old intuitions:
Once an AI can engage in targeted self-modification this capacity will unlock some off trend acceleration to capabilities improvement
Currently AIs are targeted at arbitrary cognitive tasks, but they can eventually be targeted more precisely at improving on questions that lead to higher payoff in terms of agency/intelligence/[other general capabilities]
Both of these are variants of the idea “Fine-grained self-awareness in a learner can unlock far more efficient learning”.
Now let’s examine them.
Once an AI can engage in targeted self-modification this capacity will unlock some off trend acceleration to capabilities improvement
Assume an AI has access to some rich interface for self-modification. Previously learning was mostly SGD or similar, but the problem faced by any learning rule remains! How do you search the parameter space, and how do you attribute credit, etc.? Why should introspective access provide more than an incremental improvement to the scaling law’s coefficient? For humans for instance, our level of introspective access is just far too weak to be able to tell us anything about neurological edits even if we had the tools to do these edits cleanly!
Currently AIs are targeted at arbitrary cognitive tasks, but they can eventually be targeted more precisely at improving on questions that lead to higher payoff in terms of agency/intelligence/[other general capabilities]
I see two sub-problems here.
(2a) Problem selection and creation: Of course, some weak version of active learning is possible! You can get calibration of an amortised model to predict which questions it ‘already knows’, and which are challenging. But what does that buy us? Again a minor speed up. To do better we need to be deeply strategic about problem selection and creation. This again sounds like an intrinsically hard problem you have to search the combinatorially large space of problems to find one that you must then recognise could develop some capacity of interest.
(2b) Are there problems which ‘directly’ target core capability latents? What would it mean for a problem to provide radically better learning signal on long-horizon agency, or IQ than another problem? Seems unlikely that there are problems which across a reasonable distribution of learners are far better than existing human curricula and questions at improving these competencies. If we want problems that are particularly valuable to an individual learner (AI), such problems exist but again as in (2a) they are intrinsically hard to find.
As an example of these phenomena, consider obstacles to improving on long horizon decision making: Situations where very long horizons matter are sparse. Opportunities to train that capability (i.e., get dense feedback on genuinely long-run plans) are also sparse. What’s more, the capacity to acquire increasingly long-horizon thinking may be generic, but particular long-horizon plans remain domain-specific.
Along the lines of your artist example, I find the instrument case to be a nice intuition pump.
An instrument is a technology that is deeply empowering! The human vocal range can simulate a vast range of sounds, but it’s very hard to do so and composing with just your voice (in the way one can having played/access to a piano) is I imagine impossible.
Another important facet of this example is that directly working with waveforms via a programming language or even with an interface, e.g. of a DAW, is universal but not empowering in the same way!
I think of this example as one of a broader range in which the interface is optimized for rich human interaction. One can imagine that in certain worlds interfaces become increasingly optimized for AI interaction. For example, future AIs likely will likely disprefer GUIs etc.
Formalizing what is meant by good vs bad interfaces may be another way to get useful notions of empowerment.
I suppose there’s two questions here:
How strong is generalization in general in RL?
Is there a ‘generalization barrier’ between easy-to-verify and hard-to-verify tasks
I’m guessing you mainly are thinking of (1) and have (2) as a special case?
To respond to your question, I’m reading it as:
We assume that there’s a constant multiplier in samples-to-performance needed to match in-domain training with out-of-domain training. For ‘nearby’ verifiable and non-verifiable tasks is that constant >= 10x?
I would guess modally somewhere 3-10x. I’m imagining here comparing training on more more olympiad problems vs some looser question like ‘Compare the clarity of these two proofs’. Of course there’s diminishing returns etc. so it’s not really a constant factor when taking a narrow domain.
I do agree that there are areas where domain-specific training is a bottleneck, and plausibly some of those are non-verifiable ones. See also my shortform where I discuss some reasons for such a need https://www.lesswrong.com/posts/FQAr3afEZ9ehhssmN/jacob-pfau-s-shortform?commentId=vdBjv3frxvFincwvz
I’ve done a quick follow up investigation looking into what’s going on with the adversarial critic and rebuttal optimization not doing much. My takeaway is that the above results obscure two roles of adding additional critic/rebuttal samples: (1) via averaging and consensus over the default critic/rebuttal distribution (2) via adversarial tilting of the distribution. The effect of (1) is large, and the effect of (2) is minimal except in the LiveCodeBench critic+rebuttal setting.
Here are the two relevant plots:
Average of
Best-of-
This is surprising to me since recent work found simple majority voting matches all fancy consensus methods. Meanwhile debate-like aggregation outperforms by ~20pp. There may be a continuum between consistency/aggregation methods and debate-like methods!