Oliver Sourbut

Karma: 1,237

Autonomous Systems @ UK AI Safety Institute (AISI)
DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
Former senior data scientist and software engineer + SERI MATS

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut 26 Nov 2025 13:54 UTC
4 points
0
on: Snippets on Living In Reality
How would more data and software live in reality?

A combination of more pervasive and reliable provenance annotation (citations, signed raw observations/recordings, trust-weighted endorsement of provenance claims, …) and discourse mapping (connecting supporting and contradictory arguments and evidence, clustering claims semantically, navigation of the emergent graph) would move some way towards this. FLF is among organisations working toward boosting societal collective intelligence by developing these sorts of tech. Many strategy and design variables to resolve! And questions of distribution/demand. But the design space looks newly feasible ^[[1]] and ripe for exploration.
1. ↩︎
  for example through massive cheap clerical labour from suitably scaffolded LM agents, and semantically-sensitive software (again via LMs).

Oliver Sourbut 18 Nov 2025 10:51 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: An Analogue Of Set Relationships For Distributions
For sure you can’t have cross-entropy constrained by both of $H_{P}$ and $H_{Q}$ if they [ETA that is, $P$ and $Q$ , not their entropies] differ at all. Maybe you’d make some headway if you introduced some slackness into the entropy constraints? Or maybe this is a deadend. Or maybe I’m not following right.

Oliver Sourbut 18 Nov 2025 10:47 UTC
4 points
0
in reply to: Oliver Sourbut’s comment on: An Analogue Of Set Relationships For Distributions
I think this reveals that the maxent problem you post at the start (maximum entropy constrained by cross-entropy not exceeding entropy) is actually fully specified by the constraint, rather than being sensitive to the expression being maximised. So the maxent is irrelevant. Don’t know what downstream implications that has.

Oliver Sourbut 18 Nov 2025 10:44 UTC
4 points
0
on: An Analogue Of Set Relationships For Distributions
Just a note, your notation $E_{P} [- log Q [X]]$ is the cross-entropy $H (P, Q)$ which indeed (Gibbs) is minimised (for given $P$ ) when $Q = P$ .

Oliver Sourbut 17 Nov 2025 10:23 UTC
2 points
0
on: Wiki AI
I sometimes (since 2022^[1]) say that I’d prefer if the widespread LM products ‘spoke in wikipedia voice’ rather than being specifically trained to put forward a facade of personhood (first person pronouns etc), which is especially geared to bypassing rational engagement. I don’t think people mostly thought of LMs as ‘the AIs’ until chat posttraining hit it big.

Sadly I think that’s by default a losing battle as the chat interface is both especially navigable (as you mention) and intuitively appealing to a mass market.
1. ↩︎
  It’s another draft post sitting in my queue for embarrassingly long

Oliver Sourbut 17 Nov 2025 10:18 UTC
3 points
0
in reply to: Oliver Sourbut’s comment on: Wiki AI
Note that we’re aware of the cautionary tales of Cyc, Xanadu, and Arbital!

We’re hoping a combination of:
- more minimalistic
- more open and interfaceable
- semantically sensitive software components (LMs)
- cheaply available clerical labour (LMs)
means that ‘this time is different’.

Oliver Sourbut 17 Nov 2025 10:10 UTC
2 points
0
on: Wiki AI
At FLF one of our initiatives we’re recruiting for is an ‘epistemic stack’, which I think fits the bill as a backend/foundation for many of the desiderata you’re describing. LLM chat interface would be one UX form factor on top.

Epistemic stack would be a (probably distributed) cache of annotations and metadata connecting claims to supporting sources, constructable and expandable dynamically. The cheapest and widest-coverage construction would use LM-based agents over webtext, inferring support from existing links, citations, and sometimes proactive web search for supporting (or contradictory) sources. Human participants (authors and readers) would be able to provide various annotations, including endorsement of (or alterations to) inferred epistemic support links. Something like git, versioned, signed, annotated DAGs would then be available to downstream epistemic applications (including via RAG for LM consumption, but also many other imaginable formats).

MVP on-demand tree construction for a given claim is already practical, though unreliable and more expensive than a system with caching would be.

Down the line, if more verifiable sources of ground data (signed cameras, etc.) get more widespread, such data would readily integrate as leaves.

Compare also Society Library which has some similar prior work (mostly manual), and may be moving in a similar direction.

There has also been related discussion of ‘wikipedia for LLMs’, and though I haven’t heard more technical refinement from such proponents, that term might be intended to expand to a similar overall concept.

Note that ‘wikipedia by LLM’, like grokipedia, does not currently have any meaningful claim to epistemic grounding, correctability, or transparency/legibility, though its form factor would at least inherit the navigability of wikipedia.

Oliver Sourbut 13 Nov 2025 11:17 UTC
5 points
0
on: Social drives 2: “Approval Reward”, from norm-enforcement to status-seeking
a 2×2 table for when

I think the friend/enemy axis probably works more like a scalar coefficient (or has many more than two categories). But otherwise I’m really interested to see you (who’ve put much more effort in here) converge on a similar model to my much rougher one. Does that add up for you, or do you think something else is happening? (I could also imagine a model with two coefficients, one positive, one negative, which could distinguish ‘frenemy’ from ‘stranger’.)
Thanks for this post! (It’s a reminder I keep wanting to read more of your stuff and not making time for it.)

Oliver Sourbut 12 Nov 2025 14:52 UTC
6 points
0
on: The problem of graceful deference
See also Deferring from Cotton-Barratt (2022)

You might defer because you think they know better (epistemic deferring), or because there is a formal or social expectation that you should go along with their view (deferring to authority).

Both types of deferring are important — epistemic deferring lets people borrow the fruits of knowledge; deferring to authority enables strong coordination. But they are two-edged.

...when someone defers without everyone understanding that’s what’s happening, it can cause issues. Similarly, unacknowledged expectations of deferral from others can cause problems.

Oliver Sourbut 10 Nov 2025 9:55 UTC
12 points
0
on: Unexpected Things that are People
My rough recollection is that in several early mesopotamian civilisations, much land (a majority?) and property was ‘owned’ by gods, for whom the various classes worked (priests administrating, peasants farming, and so on).

ETA Britannica has a discussion:

They constituted, as it were, a landed nobility, each god owning and working an estate—his temple and its lands

Oliver Sourbut 6 Nov 2025 7:46 UTC
2 points
−7
in reply to: Ruby’s comment on: Cancer has a surprising amount of detail
A bugbear of mine is being told to be surprised or astonished. Especially when I wouldn’t be, but even when I am. It’s not up to you, author! This post is riddled with that particular dark art, which I perceived to be on the rise in LW.

Oliver Sourbut 22 Oct 2025 13:22 UTC
2 points
0
in reply to: emmett’s comment on: ⿻ Symbiogenesis vs. Convergent Consequentialism

The real danger is less to “humanity” and more to specific humans. Because when multicellularity arose, some cells made it, some didn’t, and those that didn’t probably suffered greatly.

I’d quibble, where this depends strongly on the interpretation of ‘suffered’, and ‘made it’!

Oliver Sourbut 22 Oct 2025 13:20 UTC
4 points
0
in reply to: emmett’s comment on: ⿻ Symbiogenesis vs. Convergent Consequentialism
Another important question is whether (or under what conditions) it’s nice to be part of a greater whole. The kinds of primitive (?) greater wholes that humans currently get themselves into seem to vary in that regard, and one could imagine far greater variety there. Would it be nice to be a mitochondria-ish or gene-ish component of a larger being?

And one should note that, in the examples you’re giving, and perhaps in your described (if not conceived) vision, these are exclusive, hierarchical aggregations. (After all, my cells are part, exclusively, of me, and my mitochondria are part, exclusively, of a given cell, etc.)

In sharp contrast, contemporary human super-entities are very overlapping, as well as being porous, in ways which I think are very important and valuable (we often consider it a malaise if an aggregate is totalising and exclusionary, perhaps one of the bad markers of a cult—that fuzzy term).

Oliver Sourbut 22 Oct 2025 13:03 UTC
6 points
0
in reply to: emmett’s comment on: ⿻ Symbiogenesis vs. Convergent Consequentialism
I appreciate and partly agree with this perspective, but I think you’re expressing it quite a bit too strongly.

As salient counterpoints, consider that nature repeatedly (in unrelated phyla) invented ganglia and brains, and that human-invented systems sometimes benefit from distribution and parallelism while other times benefiting from centralisation and vertical scaling.

So I certainly don’t think you can say that nature empirically strenuously asserts in either direction.

I do think you’re on the money saying

There is little reason to suppose an silicon-based and carbon-based life cannot likewise form greater wholes.

though of course a key question is whether it’s competitive (and under what conditions), and if not, what can be done about that (or what can be done to set conditions so that it is).

Oliver Sourbut 15 Oct 2025 13:50 UTC
17 points
3
on: The “Length” of “Horizons”
I think the benchmark is intended to measure performance on an even narrower proxy than this—roughly, the sort of tasks involved in ordinary, everyday software engineering.
Note that METR has also published a subsequent attempt to broaden the class of activities, and has some suggestive results that the qualitative exponentially increasing time horizon phenomenon is somewhat robust, but the growth rate varies between domains.

Oliver Sourbut 14 Oct 2025 15:20 UTC
3 points
1
in reply to: Oliver Sourbut’s comment on: The Mom Test for AI Extinction Scenarios
On climate change, I just think it’s a point worth making: people are getting exercised about very minor contributions to resource consumption by current AI firms, which is a bit silly, but it is continuous with the kinds of radical and extinctive activities which might stamp out humans forever!

Oliver Sourbut 14 Oct 2025 15:20 UTC
3 points
−1
in reply to: Oliver Sourbut’s comment on: The Mom Test for AI Extinction Scenarios
On persuasion, it looks like you might have gone way too hard, if you’ve been arguing for arbitrary puppeteering out of the gate! (Though perhaps the quote in your mom’s voice is hypothetical?)

I’m also offering an example of a way in to discussion (again most workable one-to-one), pointing out the many and varied ways that humans persuade and coerce each other.

Oliver Sourbut 14 Oct 2025 15:19 UTC
3 points
−1
in reply to: Oliver Sourbut’s comment on: The Mom Test for AI Extinction Scenarios
On bio, I’m disagreeing that the mom test criteria override the importance of emphasising important interlinked issues. It’s not necessary to mention bio in all conversations, but you shouldn’t shy away from it, and it’s a very available and reasonable example to turn to of the kinds of vulnerabilities that could wipe out huge populations or even all humans.

Oliver Sourbut 14 Oct 2025 15:18 UTC
3 points
0
in reply to: Oliver Sourbut’s comment on: The Mom Test for AI Extinction Scenarios
On ‘totally outclassed’, I was just offering some ways that in conversation you can make that point relatable. It’s generally far more workable one-to-one, since you’re having a conversation. Less likely to work in writing, though maybe.

Oliver Sourbut 14 Oct 2025 15:18 UTC
3 points
2
in reply to: Taylor G. Lunt’s comment on: The Mom Test for AI Extinction Scenarios
My comment was actually (perhaps unhelpfully) a series of somewhat independent comments. I’ll fork under here. In sum, you could say I’m arguing that the identified heuristics for ‘mom test’ aren’t necessarily well fit, in part by giving reasons and angles-on-reasons which are (in my experience) more effective than those implied by the discussion you give to justify the heuristics. I’m also offering a few angles which I’ve found to be useful conversation openers.