Oliver Sourbut

Karma: 2,277

Autonomous Systems @ UK AI Safety Institute (AISI)
DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
Former senior data scientist and software engineer + SERI MATS

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut 21 Sep 2026 13:55 UTC
2 points
0
on: Data bottlenecks won’t prevent an intelligence explosion (but they will slow it down)
This is your best essay yet on this topic, directly addressing several of my dissatisfactions with previous discussions. Thank you!

I wish you’d more consistently use conditional language: ‘an intelligence explosion would...’ rather than ‘the intelligence explosion will...’.

What would you say if I said I still think there’s a good chance that meaningful intelligence explosion (accelerating absent compute growth) is difficult or impossible prior to roughly-human sample-efficiency? (Maintaining research taste over a moving frontier requires sample-efficient taste accumulation, and said taste is a critical factor.) Perhaps you also think something like this yourself? But perhaps you think the jagged bonuses of AI mean parity is achievable substantially before this point?

It’s believable to me that sample efficiency undergoes a meaningful jump at this (slightly indeterminate) stage. Still unclear if that’s an accelerating SIE/singularity. Do you consider your and others’ previous work on estimating the returns to R&D to be the key summary on this question?

I’m with you on learning algorithms generalising further than learned models. I possibly put more weight on bottlenecks to integration and in-domain data gathering than you. I remain unsure where this ends up cashing out in terms of a) hard power, b) social influence, and c) broad economic impact.

Oliver Sourbut 21 Sep 2026 8:50 UTC
2 points
0
in reply to: Hastings’s comment on: The Anatomy of a Chinese AI Researcher
Point of tentative information: the trisolaran system of the novel is relevantly three-body-like in that the fourth body you’re mentioning has negligible mass compared to the other three. (If I remember right. I read the book ten years ago.)

This matters in the same way that the ‘three body problem’ of Sun-Earth-Moon is in practice ~solved due to the relative mass ratios. Not a physicist, and I don’t deny that the novel has other accuracy issues (though I remember finding it pretty interesting).

Oliver Sourbut 18 Sep 2026 12:14 UTC
4 points
0
in reply to: Zack_M_Davis’s comment on: The Locally Optimal Discursive Posture
I’d add that acting curatorially/librarianish in sharing an earnest attempt at a ‘full picture’ strikes me as at least as high-integrity as sharing only the arguments which are principally upstream of one’s own considered position. And the defiled version of this is presenting as thorough while in fact collecting only the arguments that push in the direction you want to move people.

Oliver Sourbut 18 Sep 2026 12:10 UTC
2 points
0
on: The Locally Optimal Discursive Posture
I’ve been ticked off by encountering various of your emissions (and omissions) over the years, Dean, but this goes a long way to helping me understand where you’re coming from and the pressures you’ve faced (or perceived). I’ve been truly pleased by several of the essays of yours I’ve come across as well. The independent verification org direction is great. And your early discussion (however cryptic) of protocols for agent oversight was on the money. I don’t know why you’re at OpenAI now. It seems like the wrong place for you. It’s a very rhetorically overwhelming environment, and it lumbers you with an extremely salient conflict of interest. You’re much closer to the details; perhaps there’s a very strong case for it.

Oliver Sourbut 18 Sep 2026 9:52 UTC
9 points
3
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
Agree Vladimir’s posts on this are illuminating. See also The Goodhart Singularity (though compare Data bottlenecks won’t prevent an intelligence explosion).

My view is still uncertain, but incorporates elements of the above. I expect R&D automation is ‘taste complete’, and that maintaining frontier taste requires ceteris paribus human-level sample efficiency. So not much acceleration prior to that. But speed/scale will provide some degree of qualitative adjustment (perhaps not much?). Compute likely remains a constraining factor.

And then actually ‘moving atoms’ (as you put it) requires a bunch of task-/domain-specific data. That means (perhaps slow, faffy?) integration of sensing, log ingestion, etc.

If sample efficiency remains elusive, we might even see an industrial ‘explosion’ before an intelligence explosion, since manufacturing at existing tech levels is relatively repeatable compared with frontier-pushing R&D.

Oliver Sourbut 18 Sep 2026 9:38 UTC
4 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
As an even more visceral (IMO) ‘Shepard tone’ analogue, this time in tempo, see the rather incredible Troy track from Goransson’s Odyssey soundtrack.

Oliver Sourbut 15 Sep 2026 10:06 UTC
3 points
0
on: The Talker Does Not Control The Doer (in Current AIs)
I know some humans like this!

You can observe something which is perhaps related in contexts where ‘assistant’ talker turns can be modified. The actions subsequently taken sometimes totally ignore the injected stated intentions. This is anecdotal, but I imagine there are studies on this. Of course it’s a bit different, maybe confounded by distribution-shift. And there are also cases where such injections do change downstream behaviour; in fact this is a great jailbreaking channel.

Oliver Sourbut 15 Sep 2026 9:11 UTC
2 points
0
on: Mitigating Reward Hacking as Institutional Design
Unlike the policy model, the reward model cannot synthesize an infinite number of these trajectories and label them. It ultimately relies on human labelling at least for the edge cases, since ultimately the goal of the reward model is to distill ‘our’ ‘true reward function’ into an operational system. It is possible that each bit of information we provide generalizes much further than bits the policy model gets from the reward function, but there is nevertheless a fundamental asymmetry here.
Pedantry: I agree with something like the practical upshot of this claim, which is that it’s difficult and expensive and you might expect orders of magnitude cheaper generation of policy-vs-reward-model samples than reward-model-vs-actual samples.
And naively you can’t generate samples against the true reward without running real trajectories, which is costly (risky, time-consuming, expensive).
But I take pedantic issue with ‘cannot synthesize an infinite number..,‘. With a sufficiently carefully constructed ‘counterfactual machine’, in principle reward modelling could get an unbounded generator of information too, by querying (assisted) humans regarding acceptability/desirability on imagined trajectories or trajectory-summaries. It could proceed in conjunction with a schedule of adaptive sampling (probing for high-uncertainty imagined trajectories). That requires decent world modelling of some sort to get the counterfactual imagined trajectories and consequences. In the limit it might instantiate misaligned counterfactual policies which are competent enough to ‘hack out’ of the counterfactual machine, so there’s that residual concern still. And of course there’s the concern that the human raters would make mistakes, or that the counterfactuals miss the real distribution importantly, or that summaries miss/hide important details, …
So in a cardinal sense I’d say these generative processes are ‘just as infinite’ as each other, but (to defuse my point somewhat) kind of the whole point of reward modelling is so that you can make the policy-to-reward-model trajectories far denser than you can afford with ‘real live’ feedback.

Oliver Sourbut 12 Sep 2026 22:04 UTC
4 points
0
on: Some ways AI could kill us all

Chimpanzees would not imagine guns. They would not foresee poison gas. They would not conceive of chemical castration. They would not imagine humans going around and intentionally infecting them with AIDS.

It’s a bit weird because all your examples of how humans might kill chimps aren’t how we kill most chimps. We just rearrange their environment and take their stuff without even thinking about them, mostly. That’s on my list.

Oliver Sourbut 4 Sep 2026 10:19 UTC
2 points
0
on: How concerned should we be about Astra’s recurrent architecture?
Presumably, adding recurrent loops will have a slower no-CoT scaling curve than adding additional layers
Why do you think this? Because they’re forced to share weights/have less capacity? But also weight-sharing is sometimes useful for generalisation. I don’t know which way it’d balance out.

Oliver Sourbut 26 Aug 2026 14:06 UTC
3 points
1
on: Twenty Years from RSI to Takeoff: Slow Learning, Scaling Slowdown, Industrial Explosion
In my queue of posts to write is ‘Industrial explosion before intelligence explosion?’ - I think you probably nailed that better here than I would! Though I might write my own spin on it anyway at some point.

Perhaps my best case against:
- my picture of what holds back AI-driven R&D progress is ‘taste’, which is determined by sample-efficient learned value-of-information modelling at the experiment frontier
  - (perhaps this corresponds to ‘slow learning’ in your discussion)
  - (taste of course also depends on exposure to outcomes of experimental throughput)
- that makes sample efficiency one of the absolutely crucial determinants of where RSI lands
- sample efficiency might be like mathematical proofs and cyber in that it’s quite straightforward to verify and experiment in silico
- if ‘straightforward to verify (in silico)’ does mean ‘easy to make progress on’, then generalisable sample-efficiency gains might be the kind of thing AI can automate sooner rather than later

Oliver Sourbut 21 Aug 2026 8:50 UTC
3 points
0
in reply to: 2001zhaozhao’s comment on: The Rogue Agent Explosion Will Be Mostly Invisible
I have a similar concern, but this proves too much: human society (in functioning societies) manages to make the costs of making criminal livings sufficiently unappealing that it’s marginalised.

Sometimes that includes the alternative (licit activity) being more appealing (that’s ‘agent society’ in your terms I think?). Other times it means proactively deterring, disabling, and punishing activity outside the condoned structures (especially, but not only, if it’s risen to harmful levels).

Automated cyber forensics? Agent privateering? Agent bounty hunting?

Oliver Sourbut 21 Aug 2026 8:45 UTC
2 points
0
in reply to: lilkim2025’s comment on: The Rogue Agent Explosion Will Be Mostly Invisible
Government-protected and -backed criminals are an important case.

Leaving that aside, I think there’s a point at which a ‘mastermind human’ becomes a liability (less portable, physical points of intervention, expensive upkeep, …). Of course, ‘enabler humans’ (witting or otherwise) -- with analogy to organised crime and their grey-area enablers (laundering etc.) -- might be helpful long after that point.

On net I’ve been unsure how to think about this for a few years at this point. I wouldn’t confidently make the ‘strategy stealing’ kind of claims you’re making, in particular because I expect there to be comparative advantages that untethered or weakly-tethered agents have, and some of those may involve crime.

Oliver Sourbut 18 Aug 2026 8:28 UTC
2 points
0
in reply to: Cat McGee’s comment on: What gives you away: how LLMs form opinions of you
I’m a rookie re interp, but I would think that the baseline/bias is a property of the probe (since the bias would fit the probe’s training dataset?) rather than of the model being probed? (So the probe gives a good reading of the directional effects but not necessarily the absolute reading?)

Oliver Sourbut 10 Aug 2026 14:00 UTC
2 points
0
on: On Democratizing ASI to Preserve Civil Liberties
One version of a good-ish trajectory I imagine has something like a step 1.b to defuse the catastrophic misuse problem, which involves sufficient foresight and exploratory engineering to anticipate particularly disruptive or destructive prospects, and 2.a, checking in on development projects to make sure they’re either foreclosing those or delayed enough to 2.b, develop countermeasures in the mean time.

Failure modes also left as an exercise to the reader.

Oliver Sourbut 10 Aug 2026 12:56 UTC
2 points
0
on: Is it ethical to work on general-purpose robots given the risk of totalitarianism?
I’d say there’s been some discussion, for example these, all from 2025:
- acute: AI-enabled coups, Davidson et al
- gradual: Gradual Disempowerment, Kulveit and Douglas et al
- possibly: Law-Following AI, O’Keefe et al (discussion of ‘henchmen’, I can’t remember if they go into robots)
(and various discussions following those, as well as earlier discussions which are less salient in my mind right now).

Gist is basically what cousin_it says: oof, scary, because it could directly surveil+oppress, and because it erodes widespread human leverage-via-usefulness (as workers or soldiers), and because in replacement contexts it could throw whistleblowing and qualms out the window with the displaced human (these are all interlinked).

General-purpose isn’t necessarily the crucial matter, but obviously makes all of the above more straightforward and obvious.

Oliver Sourbut 3 Aug 2026 10:09 UTC
2 points
0
in reply to: Adrià Garriga-alonso’s comment on: Conditions for mathematical equivalence of Stochastic Gradient Descent and Natural Selection
Right on both counts, thanks!

When writing this I assumed there would be prior art but didn’t come across it. That link you shared looks very related and interesting, thanks.

I would have sent many mutations in all directions and then made the selection distribution really sharp based on fitness.

Many such variations possible; the spirit comes out the same under the right limiting assumptions.

Oliver Sourbut 26 Jul 2026 12:39 UTC
2 points
0
in reply to: Noosphere89’s comment on: LLMs are (still) mostly powered by imitative learning, not RL
Right, so you think the weightings won’t move sufficiently toward interactive/RL curricula for another few years at least? I agree that would preserve some incentive (not quite a constraint) toward keeping externalised CoT in token space rather than embeddings, at which point there’s at least a reasonable chance of that remaining broadly legible.

I’m not sure compute vs data is the way I’d frame this; rather I’d say that the ‘fossil fuel’ of crudely curated scraped and compiled data still has some runway ^[1] (which weakly constrains toward unsupervised pretraining and externalised CoT architectures as we’ve discussed). If data pipelines start getting more synthetic/RL-heavy, creation/processing/learning of that is still compute-constrained.

I honestly don’t know what to forecast in terms of reliance on one or other source of data. There’s probably still a lot of room for dedicated curation of non-interactive curricula, or a kind of partially-interactive (but not live RL) approach like DAgger. But interactive environments and RL are easier to scale and turn the crank on once set up.
1. ↩︎
  How much runway? Don’t know!

Oliver Sourbut 24 Jul 2026 19:38 UTC
10 points
0
on: Democracy isn’t ready for the AI revolution
Are you familiar with the gradual disempowerment framing, the (other) work of the Alignment of Complex Systems group, the intelligence curse framing, or Forethought’s work on seizing of power?

This post is a nice, approachable (in my view) discussion which seems to line up closely with some themes in those works.

Oliver Sourbut 24 Jul 2026 18:27 UTC
4 points
0
in reply to: Steven Byrnes’s comment on: LLMs are (still) mostly powered by imitative learning, not RL
Yeah, for what it’s worth, it’s disappointing to me how heavily interpretability-loaded (CoT included) many leading safety plans are, and I’m cynical about that for other reasons (like “people just won’t really bother looking at the CoT much anyway”, gestured in the conclusion of that post).

Thanks.