Oliver Sourbut

Karma: 1,310

Autonomous Systems @ UK AI Safety Institute (AISI)
DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
Former senior data scientist and software engineer + SERI MATS

I’m particularly interested in sustainable collaboration and the long-term future of value. I’d love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read—let me know your suggestions! In no particular order, here are some I’ve enjoyed recently

Ord—The Precipice
Pearl—The Book of Why
Bostrom—Superintelligence
McCall Smith—The No. 1 Ladies’ Detective Agency (and series)
Melville—Moby-Dick
Abelson & Sussman—Structure and Interpretation of Computer Programs
Stross—Accelerando
Graeme—The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

Hanabi (can’t recommend enough; try it out!)
Pandemic (ironic at time of writing...)
Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who’ve got to know me only recently are sometimes surprised to learn that I’m a pretty handy trumpeter and hornist.

Oliver Sourbut 21 Dec 2025 19:52 UTC
5 points
2
in reply to: Eliezer Yudkowsky’s comment on: Contradict my take on OpenPhil’s past AI beliefs
Makes sense. Toby Ord? Does Anders count? Or the actual Bostrom? I think that crowd did better than OP by quite a bit, and the wider Oxford AI safety community was quite good. I only met them months before ChatGPT. Will seems still surprisingly (from my view over-) optimistic, but is doing some pretty relevant and good work right now, and is usually careful to caveat where his optimistic assumptions are loadbearing.

Oliver Sourbut 21 Dec 2025 7:43 UTC
4 points
0
on: Contradict my take on OpenPhil’s past AI beliefs
I think mainly you’re asking about OP in particular, but a side question:

Who is ‘Oxford EA’? I definitely interacted with many Oxford-based EA(-adjacent) people, though only since 2022 (pre chatGPT), and the range of views and agendas was broad, and included ‘AI soon, very deadly’. I’d guess you mean some more specific (perhaps a funding- or otherwise-rich) smaller group, and I can believe that earlier views were differently distributed.

Oliver Sourbut 21 Dec 2025 1:00 UTC
3 points
0
in reply to: rvnnt’s comment on: A Full Epistemic Stack: Knowledge Commons for the 21st Century
I agree strongly with the benefit of fluent, trustworthy (perhaps customised/contextualised) summarisation. I think LMs are getting there with this, and we should bank on (and advocate/work for) improvements to that kind of capability. Probably costly right now to produce things bespoke, but amortising that by focusing on important, wide-reach content might be quite powerful.

Part of the motive for the discussion here of structure mapping (inference and discourse) is that this epistemic structure metadata can be relatively straightforward to validate, just very time consuming for humans to do. But once pieced together, it should offer useful foundation for all sorts of downstream sense making (like the summarisation you’re describing here).

Oliver Sourbut 21 Dec 2025 0:54 UTC
3 points
0
in reply to: rvnnt’s comment on: A Full Epistemic Stack: Knowledge Commons for the 21st Century
It’s difficult—the main technical thing is probably identifying (important) non fakes, rather than identifying fakes, which seems very difficult (though maybe another useful layer). The main two mechanisms I think are reputation-based (the point being to make it costly or impossible to fake a reputation for honesty) and cryptographic-based (the point being you can prove A didn’t produce X by proving that H produced X).

Oliver Sourbut 21 Dec 2025 0:29 UTC
3 points
0
in reply to: rvnnt’s comment on: A Full Epistemic Stack: Knowledge Commons for the 21st Century
Oh, at this stage yes I’m reasonably sure it’d be technically feasible for those criteria to be operationalised in a mostly not-too-lossy way.

The difficulty is indeed primarily in distribution and in ideals meeting people where they are. (There’d also be some question of token expense if you were running contextually sensitive smarts on every single post, though that continues to get cheaper.)

We’ve used the term ‘aligned recommender systems’ to capture the idea that content feeds and information diets could be much more edifying and so on. Trying to be too preachy and prescriptive is presumably doomed to fail. Perhaps an incentive-compatible middle way is to hook into people’s more reflective endorsement signals, and better nuanced feedback than clicks and other engagements.

Oliver Sourbut 19 Dec 2025 11:45 UTC
2 points
0
on: A name for the things that AI companies are building
‘Agentic’ or ‘agent’ is getting a fair bit of currency (‘agentic AI workflow’, ‘LM agent’, ‘AI agent’, etc.)

I think that’s fine, and basically accurate. Sometimes it means you need to qualify how autonomous or bounded/unbounded the looping is.

‘Model’ really gets my goat, is a terrible, already hopelessly conflated term, and should be banned for talking about NNs in almost all contexts. (I have been dragged kicking and screaming into using this sometimes and I’m still sad about it.) ‘Reasoning model’ is no better. ‘Foundation model’ and ‘language model’ are OK, but only if actually talking about foundation and/or language models per se, absent the various finetuning and scaffoldings that are involved in actual AI systems. (‘Reward model’ and ‘world model’ and such are very reasonable uses.)

I’m sorry to say that ‘neuro-scaffold’ isn’t going to take off, and I think that’s fine. ‘Scaffold’ is very useful on its own, but ‘neuro-scaffold’ is a mouthful and also doesn’t really connote the specific thing you’re meaning to invoke, which is the loopiness and the connection to actuators.

Oliver Sourbut 17 Dec 2025 14:43 UTC
4 points
−2
in reply to: Jan_Kulveit’s comment on: A Case for Model Persona Research
I’m interested to know how (if at all) you’d say the perspective you’ve just given deviates from something like this:
- pretrained LMs can simulate many personas, including interpolations and extrapolations of personas inferred from the training data
  - (aside: this presumably also includes various non-first-person non-‘self’-representing ‘personas’ like wikipedia voice and stackoverflow voice)
- various forms of conditioning can elicit/filter personas
  - finetuning of all sorts
  - also prompting (e.g. playscript-style)
  - (this can include autoregressive ‘self’ prompting)
- depending on type and level of finetuning, weights can end up more or less cohering on a particular subset/subspace of personas
  - (this might be context-dependent, for example if certain prompt conditions, such as a user/assistant playscript, are consistently present in the finetuning, and might not generalise to other prompt conditions)
My current guess is you agree with some reasonable interpretation of all these points. And maybe also have some more nuance you think is important?

Oliver Sourbut 17 Dec 2025 13:47 UTC
2 points
0
in reply to: Oliver Sourbut’s comment on: Slow corporations as an intuition pump for AI R&D automation
Given the picture I’ve suggested, the relevant questions are
- What are the returns to thinking longer for experiment design?
  - (I tentatively think they’re quite harsh)
- What are the penalties for parallelised experiment design and/or for teamwork on experiment design?
  - (I’ve written even less on this, but I think it’s quite difficult because the key is pooling research taste, either from best-of-k or some team iterative thing, but research taste is mostly acquired through experimental experience)
- (Perhaps also as you note, are there cognitive diversity penalties or bonuses at play?)

Oliver Sourbut 17 Dec 2025 13:39 UTC
2 points
0
on: Slow corporations as an intuition pump for AI R&D automation
In my somewhat analogous picture (which I’ve thought through on paper less than you), having AI workforce is more like
- having more people
- who can effectively think for longer between experiments
(The inverted slowcorp would have fewer people who are more distracted/have less time to think between experiments.)

To me that’s the natural operationalisation of this analogy. I notice it doesn’t obviously map directly to your analogy, where there are different quantities of compute, different serial research times, etc. Do you think your analogy is more natural, or does it in fact map more closely than I’m seeing?

Oliver Sourbut 15 Dec 2025 17:47 UTC
4 points
2
in reply to: metachirality’s comment on: Oliver Sourbut’s Shortform
I think unless they explicitly want to harm or threaten you, was the point—which incidentally is often a situation not accounted for in the foundational assumptions of many economic models (utility functions generally considered to be independent and monotonic in resources and so on).

Oliver Sourbut 15 Dec 2025 17:45 UTC
2 points
0
in reply to: J Bostock’s comment on: Oliver Sourbut’s Shortform
Uhh, not sure. I think it was just a holistic consideration including the property.

Oliver Sourbut 15 Dec 2025 11:31 UTC
2 points
0
in reply to: J Bostock’s comment on: Oliver Sourbut’s Shortform
See other thread where we discuss a variant without forced sale, but with forced re-evaluation.

I don’t think anyone would advocate a naive universalist approach to this with square inches for sale. This does point at an interesting mereological dynamic here though where the boundaries of ‘an item’ might in principle be subject to some gerrymandering!

I do think that a large part of the point is to incentivise a high (somewhat Georgist) tax on ‘counterproductive’ uses of land (or similar), if I’ve read correctly.

I happen to know a former low-level land evaluation bureaucrat in the UK (for council tax) and he said it’s a huge pain. For annoying bureaucratic activities, two standard questions are ‘can we sensibly decentralise?’ and ‘can we sensibly automate?’… maybe so.

Oliver Sourbut 12 Dec 2025 19:34 UTC
2 points
0
in reply to: Dagon’s comment on: Oliver Sourbut’s Shortform
Seems right! ‘studies’ uplifts ‘design’ (either incremental or saltatory), I suppose. For sure, the main motivation here is to figure out what sorts of capabilities and interventions could make coordination go better, and one of my first thoughts under this heading is open librarian-curator assistive tech for historic and contemporary institution case studies. Another cool possibility could be simulation-based red-teaming and improvement of mechanisms.

If you have any resources or detailed models I’d love to see them!

Oliver Sourbut 12 Dec 2025 12:20 UTC
4 points
0
on: Oliver Sourbut’s Shortform
What constitutes cooperation? (previously)

Because much in the complex system of human interaction and coordination is about negotiating norms, customs, institutions, constitutions to guide and constrain future interaction in mutually-preferable ways, I think:
1. ‘institution and constitution design’
deserves special attention.

This is despite it perhaps (for a given ‘coordination moment’) being in theory reducible to preference elicitation or aggregation, searching outcome space, negotiation, enforcement, …

There are failure modes (unintended consequences, concentration of power, lost purposes, corruptability, poor adaptability, plain old inefficacy) and patterns for success (stabilising win-win equilibria, reducing inefficiencies, improving collective intelligence and adaptability) which are specific to this process of negotiating and developing institutions (there are patterns, because the complex system has emergent structure like trust, corruption, coalitions, information propagation, …).

Said briefly: much (most?) coordination is about coordination because a) humans are that type of creature and b) we live in a highly iterated world.

Oliver Sourbut 12 Dec 2025 12:08 UTC
2 points
0
in reply to: Jay Bailey’s comment on: Oliver Sourbut’s Shortform
Ideally you would want to allow depreciation though, which is a definite phenomenon! (Especially if things are neglected.)

Yeah, there’s some design questions. You’re right, the upside to the corrective bidders is naively nothing if they get called on it: they’re doing valuable corrective cybernetic labour for free.

Maybe a sensible refinement would be for them to be owed a small fee… or roughly equivalently some (temporary) direct share of the resulting increased Harberger tax.

Oliver Sourbut 11 Dec 2025 20:05 UTC
5 points
0
in reply to: Oliver Sourbut’s comment on: Oliver Sourbut’s Shortform
A less crazed approach might be more like
- if someone bids at or above your assessed value, you have to sell or start paying commensurably more tax (possibly backdated some amount)
What links here?
- Oliver Sourbut's comment on Oliver Sourbut’s Shortform by Oliver Sourbut (15 Dec 2025 11:31 UTC; 2 points)

Oliver Sourbut 11 Dec 2025 19:59 UTC
3 points
0
in reply to: homosapien97’s comment on: Oliver Sourbut’s Shortform
By this, do you mean something like: when I purchase a mine or whatever, I’m speculatively pricing in some upside (e.g. a new use) which is part of my valuation for it, and if later a marginally more alert person buys me out because of a new actual use before I update my valuation, I fail to realise that value? But if no new actual use comes up, I’m left holding the bag? I agree. And possibly we also agree that’s the same issue as the umbrella, where someone noticed it’s raining before I did?

Oliver Sourbut 11 Dec 2025 19:20 UTC
3 points
0
in reply to: homosapien97’s comment on: Oliver Sourbut’s Shortform
Leaving aside delays, this does get at a point I noticed wasn’t obviously addressed in that paper which is what to do about very seasonal things. The example I thought of (rather less macabre) was an umbrella in a rainstorm. I don’t think it’s sensibly applicable to most personal property.

Oliver Sourbut 11 Dec 2025 12:34 UTC
16 points
4
on: Oliver Sourbut’s Shortform
Learned about ‘Harberger tax’ recently.

The motivation is like
- private assets (private use and excludable) incentivise investment/productive improvement/development well
- but private ownership of non-liquid, non-fungible, idiosyncratic things (factories, houses, natural resources maybe, AI systems maybe?) acts like a monopoly, distorting allocation (more productive/valued uses aren’t often realised)
  - e.g. NIMBYism
  - e.g. ‘holdout pricing’
  - e.g. inefficient bargaining due to information asymmetry
- common ownership and lease auctioning incentivises efficient allocation
The pitch is like, can we do something in between?
- self-assessment of value (reveals something close to honest valuation)
- pay some rate of tax on that self-assessment
- you have to sell at that self-assessed value if someone wants to buy
They claim this keeps most of the benefit of investment incentivisation, because things are mostly private in practice, but substantially improves allocative efficiency by lubricating the more valuable trades.

Anyway, mainly it’s interesting because getting into the gubbins of particular proposals helps me learn about the relevant dynamics in general, but also I wondered if there’s something in this vicinity that could work nicely for AI development and AI deployment. (Like maybe a Harberger tax on compute, or on AI systems, or...)

Oliver Sourbut 4 Dec 2025 11:29 UTC
2 points
0
in reply to: habryka’s comment on: Relitigating the Race to Build Friendly AI
Yudkowsky’s 2008 AI as a Positive and Negative Factor in Global Risk is a pretty good read, both for the content (which is excellent in some ways and easy to critique in others), and for the historical interest (where it’s useful to litigate the question of what MIRI was aiming at around then, and because it’s interesting how much dynamic Yudkowsky anticipated/missed, and because it’s interesting to inhabit 2008 for a bit and update on empirical observations since then).