Oliver Sourbut

Karma: 1,276

Autonomous Systems @ UK AI Safety Institute (AISI)
DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
Former senior data scientist and software engineer + SERI MATS

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut 12 Dec 2025 19:34 UTC
2 points
0
in reply to: Dagon’s comment on: Oliver Sourbut’s Shortform
Seems right! ‘studies’ uplifts ‘design’ (either incremental or saltatory), I suppose. For sure, the main motivation here is to figure out what sorts of capabilities and interventions could make coordination go better, and one of my first thoughts under this heading is open librarian-curator assistive tech for historic and contemporary institution case studies. Another cool possibility could be simulation-based red-teaming and improvement of mechanisms.

If you have any resources or detailed models I’d love to see them!

Oliver Sourbut 12 Dec 2025 12:20 UTC
4 points
0
on: Oliver Sourbut’s Shortform
What constitutes cooperation? (previously)

Because much in the complex system of human interaction and coordination is about negotiating norms, customs, institutions, constitutions to guide and constrain future interaction in mutually-preferable ways, I think:
1. ‘institution and constitution design’
deserves special attention.

This is despite it perhaps (for a given ‘coordination moment’) being in theory reducible to preference elicitation or aggregation, searching outcome space, negotiation, enforcement, …

There are failure modes (unintended consequences, concentration of power, lost purposes, corruptability, poor adaptability, plain old inefficacy) and patterns for success (stabilising win-win equilibria, reducing inefficiencies, improving collective intelligence and adaptability) which are specific to this process of negotiating and developing institutions (there are patterns, because the complex system has emergent structure like trust, corruption, coalitions, information propagation, …).

Said briefly: much (most?) coordination is about coordination because a) humans are that type of creature and b) we live in a highly iterated world.

Oliver Sourbut 12 Dec 2025 12:08 UTC
2 points
0
in reply to: Jay Bailey’s comment on: Oliver Sourbut’s Shortform
Ideally you would want to allow depreciation though, which is a definite phenomenon! (Especially if things are neglected.)

Yeah, there’s some design questions. You’re right, the upside to the corrective bidders is naively nothing if they get called on it: they’re doing valuable corrective cybernetic labour for free.

Maybe a sensible refinement would be for them to be owed a small fee… or roughly equivalently some (temporary) direct share of the resulting increased Harberger tax.

Oliver Sourbut 11 Dec 2025 20:05 UTC
5 points
0
in reply to: Oliver Sourbut’s comment on: Oliver Sourbut’s Shortform
A less crazed approach might be more like
- if someone bids at or above your assessed value, you have to sell or start paying commensurably more tax (possibly backdated some amount)

Oliver Sourbut 11 Dec 2025 19:59 UTC
3 points
0
in reply to: homosapien97’s comment on: Oliver Sourbut’s Shortform
By this, do you mean something like: when I purchase a mine or whatever, I’m speculatively pricing in some upside (e.g. a new use) which is part of my valuation for it, and if later a marginally more alert person buys me out because of a new actual use before I update my valuation, I fail to realise that value? But if no new actual use comes up, I’m left holding the bag? I agree. And possibly we also agree that’s the same issue as the umbrella, where someone noticed it’s raining before I did?

Oliver Sourbut 11 Dec 2025 19:20 UTC
3 points
0
in reply to: homosapien97’s comment on: Oliver Sourbut’s Shortform
Leaving aside delays, this does get at a point I noticed wasn’t obviously addressed in that paper which is what to do about very seasonal things. The example I thought of (rather less macabre) was an umbrella in a rainstorm. I don’t think it’s sensibly applicable to most personal property.

Oliver Sourbut 11 Dec 2025 12:34 UTC
14 points
4
on: Oliver Sourbut’s Shortform
Learned about ‘Harberger tax’ recently.

The motivation is like
- private assets (private use and excludable) incentivise investment/productive improvement/development well
- but private ownership of non-liquid, non-fungible, idiosyncratic things (factories, houses, natural resources maybe, AI systems maybe?) acts like a monopoly, distorting allocation (more productive/valued uses aren’t often realised)
  - e.g. NIMBYism
  - e.g. ‘holdout pricing’
  - e.g. inefficient bargaining due to information asymmetry
- common ownership and lease auctioning incentivises efficient allocation
The pitch is like, can we do something in between?
- self-assessment of value (reveals something close to honest valuation)
- pay some rate of tax on that self-assessment
- you have to sell at that self-assessed value if someone wants to buy
They claim this keeps most of the benefit of investment incentivisation, because things are mostly private in practice, but substantially improves allocative efficiency by lubricating the more valuable trades.

Anyway, mainly it’s interesting because getting into the gubbins of particular proposals helps me learn about the relevant dynamics in general, but also I wondered if there’s something in this vicinity that could work nicely for AI development and AI deployment. (Like maybe a Harberger tax on compute, or on AI systems, or...)

Oliver Sourbut 4 Dec 2025 11:29 UTC
2 points
0
in reply to: habryka’s comment on: Relitigating the Race to Build Friendly AI
Yudkowsky’s 2008 AI as a Positive and Negative Factor in Global Risk is a pretty good read, both for the content (which is excellent in some ways and easy to critique in others), and for the historical interest (where it’s useful to litigate the question of what MIRI was aiming at around then, and because it’s interesting how much dynamic Yudkowsky anticipated/missed, and because it’s interesting to inhabit 2008 for a bit and update on empirical observations since then).

Oliver Sourbut 4 Dec 2025 11:23 UTC
2 points
0
on: Relitigating the Race to Build Friendly AI

improving AI strategic competence (relative to their technological abilities) may be of paramount importance (so that they can help us with strategic thinking and/or avoid making disastrous mistakes of their own), but this is clearly even more of a double-edged blade than AI philosophical competence

I think you can get less of the tradeoff here by explicitly and deliberately aiming for AI ‘tools’ for improving human (group) strategic competence. It sounds subtle, but I think it has quite different connotations and implications for what you actually go and do!

Oliver Sourbut 4 Dec 2025 11:09 UTC
2 points
0
on: Categorizing Selection Effects
(Interested to know if you have a taxonomy here which you feel is approaching MECE? I think this is more like a survey—which is still helpful.)

Oliver Sourbut 4 Dec 2025 11:07 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: Categorizing Selection Effects
p.s. I agree your ‘Agentic’ and ‘Process’ are the same thing, and I expected ‘Agentic’ to be what I just called ‘Adversarial’ (which I think is a better name for it).

Oliver Sourbut 4 Dec 2025 11:06 UTC
4 points
0
on: Categorizing Selection Effects
Adversarial selection? i.e. another wants X, believes they’ll get more X if I believe A, believes I’ll believe A if I see O, shows me O.

Oliver Sourbut 4 Dec 2025 11:00 UTC
5 points
0
on: 6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa
A complementary angle: we shouldn’t be arguing over whether or not we’re in for a rough ride, we should be figuring out how to not have that.

I suspect more people would be willing to (both empirically and theoretically) get behind ‘ruthless consequentialist maximisers are one extreme of a spectrum which gets increasingly scary and dangerous; it would be bad if those got unleashed’.

Sure, skeptics can still argue that this just won’t happen even if we sit back and relax. But I think then it’s clearer that they’re probably making a mistake (since origin stories for ruthless consequentialist maximisers are many and disjunctive). So the debate becomes ‘which sources of supercompetent ruthless consequentialist maximisers are most likely and what options exist to curtail that?’.

Oliver Sourbut 4 Dec 2025 10:58 UTC
LW: 5 AF: 2
0
AF
on: 6 reasons why “alignment-is-hard” discourse seems alien to human intuitions, and vice-versa
The human intuition that treating other humans as a resource to be callously manipulated and exploited, just like a car engine or any other complex mechanism in their environment, is a weird anomaly rather than the obvious default
e.g. as is the typical human response to people who are far away (both physically and conceptually, so whose approval isn’t salient or anticipated) i.e. ‘the outgroup’

Oliver Sourbut 2 Dec 2025 10:38 UTC
4 points
1
on: Metric-haven (quick stats on how Inkhaven impacted LessWrong)
For what it’s worth, I’ve considered unsubscribing from several inkhaven participants (on LW and elsewhere) because of the low quality, though I think I endorse the ‘practice form intensely’ manoeuvre in general and feel some sort of ‘praxis envy’/inspiration. In practice I have mostly skimmed or ignored some posts, read others, and hope that the participants’ writing will be better for it in future. I’m definitely not representative of the general internet denizenry, though I may be more representative of LW audience (and perhaps even more so LW target audience).

Oliver Sourbut 1 Dec 2025 11:17 UTC
4 points
2
in reply to: Ruby’s comment on: Why people like your quick bullshit takes better than your high-effort posts
I appreciate that this message has the germ of ‘...and this is an abominable status quo’, Ruby (even if you wouldn’t put it that way).

Oliver Sourbut 26 Nov 2025 13:54 UTC
6 points
0
on: Snippets on Living In Reality
How would more data and software live in reality?

A combination of more pervasive and reliable provenance annotation (citations, signed raw observations/recordings, trust-weighted endorsement of provenance claims, …) and discourse mapping (connecting supporting and contradictory arguments and evidence, clustering claims semantically, navigation of the emergent graph) would move some way towards this. FLF is among organisations working toward boosting societal collective intelligence by developing these sorts of tech. Many strategy and design variables to resolve! And questions of distribution/demand. But the design space looks newly feasible ^[[1]] and ripe for exploration.
1. ↩︎
  for example through massive cheap clerical labour from suitably scaffolded LM agents, and semantically-sensitive software (again via LMs).

Oliver Sourbut 18 Nov 2025 10:51 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: An Analogue Of Set Relationships For Distributions
For sure you can’t have cross-entropy constrained by both of $H_{P}$ and $H_{Q}$ if they [ETA that is, $P$ and $Q$ , not their entropies] differ at all. Maybe you’d make some headway if you introduced some slackness into the entropy constraints? Or maybe this is a deadend. Or maybe I’m not following right.

Oliver Sourbut 18 Nov 2025 10:47 UTC
4 points
0
in reply to: Oliver Sourbut’s comment on: An Analogue Of Set Relationships For Distributions
I think this reveals that the maxent problem you post at the start (maximum entropy constrained by cross-entropy not exceeding entropy) is actually fully specified by the constraint, rather than being sensitive to the expression being maximised. So the maxent is irrelevant. Don’t know what downstream implications that has.

Oliver Sourbut 18 Nov 2025 10:44 UTC
4 points
0
on: An Analogue Of Set Relationships For Distributions
Just a note, your notation $E_{P} [- log Q [X]]$ is the cross-entropy $H (P, Q)$ which indeed (Gibbs) is minimised (for given $P$ ) when $Q = P$ .