Oliver Sourbut

• Oliver Sourbut 13 Mar 2026 11:19 UTC

  9 points

  3

  in reply to: Elizabeth’s comment on: Eliz­a­beth’s Shortform

  Hmm. One of the most emotionally (?) difficult parts of deciding to leave the govt for me was knowing I’d give up my security clearance even though I’d already inferred or known ahead of time every supposedly ‘secret’ thing I’d been shown! (My clearance was not especially high level.)

  This was a new kind of flinch to experience, but I didn’t notice or reflect on the psychological generalisations and neighbours it might have at the time. Thanks for the nudge.

• Oliver Sourbut 11 Mar 2026 13:39 UTC

  2 points

  0

  on: Eco­nomic effi­ciency of­ten un­der­mines so­ciopoli­ti­cal autonomy

  I saw few mainstream economists take them into account when evaluating Trump’s tariffs

  Oh, my impression (not very studied!) was that nearly the whole conversation was about the tradeoff between efficiency (and who pays) and (re)onshoring. (The other part being the dynamic use as tactical economic weaponry.)

• Oliver Sourbut 11 Mar 2026 13:01 UTC

  2 points

  −2

  in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform

  nit: Christiano operationalised ‘slow takeoff’ via ‘world product’, not GDP. I’m not sure exactly what he meant by that (or if he had a more concrete operationalisation), but it does strike me as wise to not anchor to GDP which is awfully fraught and misleadingly conservative.

  ETA: fake news! I checked and while he starts talking about ‘output’, he later seems to operationalise it as GDP specifically

• Oliver Sourbut 4 Mar 2026 17:04 UTC

  2 points

  0

  on: I’m con­fused by the change in the METR trend

  My rough model of what’s going on (not published) is that ‘relevant effective evidence’ is what’s needed to succeed at a given subtask (of the cognitive kind that AI agents are being tested on here).

  Relevant effective evidence is accumulated in pretraining (data is information is evidence!) as well as through in-context evidence-gathering activities ^[1] .

  Generic webdata and the earlier corpora had some applicable data for these sorts of tasks. More recently, a greater fraction of the curated pretraining and posttraining data are dedicated to software engineering.

  (It’s complicated by changes to posttraining, but certainly ‘effective training data’ were increasing roughly exponentially with training date for some time, which maps to the exponential ‘time horizon’ via subtask completion chance.)

  This is all kind of annoying retrodiction but it adds up, at least in hindsight.

  1. ↩︎

     In-context evidence-gathering (i.e. exploration) competence comes from particularly generalisable learned heuristics, namely how best to experiment or try things out and interpret findings, but is still largely domain-specific.

• Oliver Sourbut 26 Feb 2026 9:10 UTC

  6 points

  0

  in reply to: Raemon’s comment on: Open sourc­ing a browser ex­ten­sion that shows when peo­ple are wrong on the internet

  If you want galaxy brain, check out my and Ben’s take :L (I’m heartened to see a fair bit of (independent?) convergence!)

• Oliver Sourbut 24 Feb 2026 12:13 UTC

  6 points

  2

  in reply to: Jan_Kulveit’s comment on: The per­sona se­lec­tion model

  Seems more than 2 years perhaps? (It’s (early) 2026!) Also I wouldn’t call this topically (though perhaps de facto socially ^[1] ) cyborgism.

  1. ↩︎

     Janus being among the earliest to write about these topics in detail

• Oliver Sourbut 24 Feb 2026 11:29 UTC

  4 points

  2

  on: The per­sona se­lec­tion model

  “What would the Assistant do?” (according to the beliefs of the post-trained LLM simulating the Assistant).

  nit: I think you’re slightly falling foul of the anthropomorphism you’re aiming to avoid, here. Rather, ‘according to the post-trained LLMs’s posterior over Assistant personas (further conditioned by the preceding context)’.

  More generally, a license to consider some things somewhat anthropomorphically (i.e. Assistant personas under certain training regimes) needs to be handled very carefully given known human reasoning biases here!

  You’re maybe falling into a similar trap in the AI welfare section (perhaps conflating LM and Assistant and Assistant-instance beliefs?) but it’s pretty tricky territory!

• Oliver Sourbut 23 Feb 2026 16:08 UTC

  3 points

  0

  in reply to: Nathan Young’s comment on: Nathan Young’s Shortform

  One naive quick response: it’s a coordination problem. Sticking your head above the parapet alone gets you shot. And you only collect a very small fraction of the public good. (Also, publicly genuflecting instead may get you favours.)

  Meanwhile, quite rightly, coordination between firms and representatives of firms is seen as a big no-no in most sectors in a good liberal democracy, and this is enforced somewhat. It’d be market-distorting!

  One of the main reasons to have a government at all is to do the coordinating in such cases, because markets can’t (be trusted to). Difficult when the government itself is the (ex hypothesi) problem.

  That’s where checks and balances, and ideally ‘the wisdom of the people’ is meant to step in. (How is that going?)

• Oliver Sourbut 20 Feb 2026 9:30 UTC

  4 points

  0

  in reply to: tailcalled’s comment on: Linch’s Shortform

  Let me check: you mean the grandchildren have the same ex ante expected height as the ex ante expected height of the children. Of course! (Just as the children have the same ex ante expected height as the parents’ ex ante expected height, which is now screened off by knowing their actual height.) But if you reset your expectations based on the observed children’s heights, you’ll still witness (on average) regression to the mean.

  (If not this, I’d appreciate an explainer!)

• Oliver Sourbut 19 Feb 2026 11:12 UTC

  6 points

  2

  on: Build­ing Tech­nol­ogy to Drive AI Governance

  Strongly endorse something like this central point: governance is often driven by cost and affordances, so the availability of observation and intervention mechanisms could be a deciding factor in governance intent. (This makes affordances doubly useful, multiplying with existing intent while also unlocking more intent/​will.)

  (Foresightful governance intent can also conjure more affordance into being by resourcing and encouraging—or even requiring—work on it.)

  You also maybe half-say this: costs can include much more than financial. Side-effects, collateral, etc. can also be major costs (especially in governance). This makes e.g. privacy-protecting innovation look appealing.

• Oliver Sourbut 19 Feb 2026 10:47 UTC

  3 points

  0

  on: Man­aged vs Un­man­aged Agency

  Very good. Some terminological thoughts:

  ‘subagent’ is quite conflated. Sometimes we’re talking about the constituent parts of an emergent group actor. Sometimes we’re talking about a delegated (perhaps created) process. Sometimes we’re talking about different threads/​personas within a constructed multi-persona reasoning scheme. There might be other cases.

  ‘agent’ is also quite conflated. Sometimes we mean an actor in a principal-agent relationship. Sometimes we mean an AI. Sometimes we mean just someone/​something acting somewhat consequentialist. Sometimes we’re talking about something closer to a naked utility function.

  ‘shard’ is ehhh. A neologism that doesn’t seem to have a fixed meaning.

  I’m somewhat thinking aloud (but have given some thought to this previously). I’d tentatively suggest using terms like:

  • ‘delegate’ for a system which is either spun up whole cloth or conditioned to go do a particular thing

    • you can qualify if wanted, like ‘delegate agent’ (which implies a fair bit of autonomy) or ‘delegate process’ (which might be e.g. a relatively more mechanical sim or something)

  • ‘subprocess’, if needed, for something that really is more like a local, non-actuating search, computation, etc.

  • ‘principal’ for the actor doing the delegating

  • (personally I want to reserve ‘subagent’ for the thing which is a constituent part of something which is reasonably thought of as a group actor, but I don’t know if that use is sufficiently Schelling to win, and expect to need different words for that)

  • only sometimes ‘agent’, but often it’s worth tabooing that

• Oliver Sourbut 16 Feb 2026 14:47 UTC

  8 points

  5

  on: Align­ing to Virtues

  Broadly agree that entraining virtues is an encouraging path, especially for agentic systems, but also for systems used as building blocks in other beneficial tech.

  I have a terminological nit: ‘aligning to’ virtue feels like a type mismatch, to me. My conception of alignment (and I think this is moderately widely held, but am interested to hear if not) is that it’s centrally about objectives, preferences, goals, etc. (and one can even literally think of it as a geometric alignment in abstract preference-space if desired).

  So to me, ‘alignment’ carries a connotation of necessarily being about preferences. I might say

  • aligning (or alignment of) objectives

  • maintaining or enforcing deontological constraints

  • entraining (or embodiment of?) virtue

• Oliver Sourbut 15 Feb 2026 10:00 UTC

  2 points

  0

  in reply to: Oliver Sourbut’s comment on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  Confused by (strong?) downvotes on this. What’s the sin? If you disagree, use the disagree buttons (and I’d love it if you explain why you disagree too)!

• Oliver Sourbut 13 Feb 2026 11:09 UTC

  13 points

  6

  on: What We Learned from Briefing 140+ Law­mak­ers on the Threat from AI

  What is disarming is the absence of performance: clear, careful speech grounded in knowledge, and an evident commitment to honesty to oneself and others.

  The interesting thing is that it seems like this should be somewhat fakeable, but in practice it seems to be quite hard for people to fake. I have found this (being earnest, honest, and expert, not being fake!) to be my weird charisma superpower that I didn’t expect. I have also found that I don’t need to hide my passion and concern once a bit of human-to-human relating has happened over food or drink.
  
  Plus 1 to all your points Leticia, keep up the good work!

  As I recently said, I sense there’s a shift now where the demand for understanding (and battle for attention) remains, but serious people are now also demanding proposals and solutions, as you’ve noted. I suspect a failure mode here could be coming from the ivory tower with talking points without being responsive to the constraints and skills and options of the lawmakers (etc.) on the receiving end. That too will need to be a dialogue, perhaps a ‘tennis’ of sorts.

• Oliver Sourbut 12 Feb 2026 21:05 UTC

  37 points

  32

  on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  I realise that ‘person-affecting’ is signposted here, but (especially as someone who considers such views to be tremendously mistaken, but generally as someone who values clear discourse), I really think it ought to be emphasised and clarified more.

• Oliver Sourbut 12 Feb 2026 20:49 UTC

  9 points

  −2

  on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  I’ve only read the abstract, intro, and conclusion, and look forward to reading in more depth later.

  I wanted to criticise the central framing which seems implicit: either we build superintelligence (which can immediately solve near-arbitrary problems in medicine, governance, and other tech), or we don’t (in which case we have no hope of becoming wiser, more capable, or better coordinated).

  This strikes me as a terribly false dichotomy, very pervasive. We can totally get really ambitious benefits from tech, including those involving AI, without the autonomy and unified generality usually connoted by ‘AGI’ and ‘superintelligence’! And there’s a virtuous cycle where each boost to our wisdom marginally reduces the risks of badly-judged gambles.

  See AI for Human Reasoning for Rationalists for a brief expansion of this (and links to more).

  Perhaps there are unconquerable obstacles (which would nonetheless be conquered by ASI) which I don’t see clearly enough. I would like to know them (I fear the proliferation of destructive means via AI, and perhaps only a sufficiently surpassing singleton, and only one generated via superintelligence, sovereign or otherwise, can overcome this).

  Or perhaps this highly distributed and human-centred approach actually counts as (some form of) superintelligence on some views—that seems reasonable on first principles, though I think colloquially confusing.

• Oliver Sourbut 12 Feb 2026 10:26 UTC

  2 points

  0

  on: The na­ture of LLM al­gorith­mic progress

  Possibly better to have a fresh new post, and point each post to the other, rather than updating in place? (You could still keep the changelog, which is polite and helpful for your readers.) In particular, I only saw this revision by luck.

Strate­gic aware­ness tools: de­sign sketches

• Oliver Sourbut 9 Feb 2026 20:13 UTC

  3 points

  0

  in reply to: rvnnt’s comment on: De­sign sketches for a more sen­si­ble world

  Yeah, I like that. For me, it comes under coordination tech, which admittedly is a particularly thorny part of the map, and difficult. Ability to network and form coalitions, deliberate on models and prospects, and implement or commit to changes come under coordination, for me. Often there’d be an aspect of institution-installment or -adaptation, which deserves special care (forthcoming ‘charter tech’ discussion, name tbc). You’d also naturally want (individual foresight and collective) epistemics to be improved as an important support to that.

De­sign sketches for a more sen­si­ble world