Oliver Sourbut

• Oliver Sourbut 4 Mar 2026 17:04 UTC

  2 points

  0

  on: I’m con­fused by the change in the METR trend

  My rough model of what’s going on (not published) is that ‘relevant effective evidence’ is what’s needed to succeed at a given subtask (of the cognitive kind that AI agents are being tested on here).

  Relevant effective evidence is accumulated in pretraining (data is information is evidence!) as well as through in-context evidence-gathering activities ^[1] .

  Generic webdata and the earlier corpora had some applicable data for these sorts of tasks. More recently, a greater fraction of the curated pretraining and posttraining data are dedicated to software engineering.

  (It’s complicated by changes to posttraining, but certainly ‘effective training data’ were increasing roughly exponentially with training date for some time, which maps to the exponential ‘time horizon’ via subtask completion chance.)

  This is all kind of annoying retrodiction but it adds up, at least in hindsight.

  1. ↩︎

     In-context evidence-gathering (i.e. exploration) competence comes from particularly generalisable learned heuristics, namely how best to experiment or try things out and interpret findings, but is still largely domain-specific.

• Oliver Sourbut 26 Feb 2026 9:10 UTC

  6 points

  0

  in reply to: Raemon’s comment on: Open sourc­ing a browser ex­ten­sion that shows when peo­ple are wrong on the internet

  If you want galaxy brain, check out my and Ben’s take :L (I’m heartened to see a fair bit of (independent?) convergence!)

• Oliver Sourbut 24 Feb 2026 12:13 UTC

  6 points

  2

  in reply to: Jan_Kulveit’s comment on: The per­sona se­lec­tion model

  Seems more than 2 years perhaps? (It’s (early) 2026!) Also I wouldn’t call this topically (though perhaps de facto socially ^[1] ) cyborgism.

  1. ↩︎

     Janus being among the earliest to write about these topics in detail

• Oliver Sourbut 24 Feb 2026 11:29 UTC

  4 points

  2

  on: The per­sona se­lec­tion model

  “What would the Assistant do?” (according to the beliefs of the post-trained LLM simulating the Assistant).

  nit: I think you’re slightly falling foul of the anthropomorphism you’re aiming to avoid, here. Rather, ‘according to the post-trained LLMs’s posterior over Assistant personas (further conditioned by the preceding context)’.

  More generally, a license to consider some things somewhat anthropomorphically (i.e. Assistant personas under certain training regimes) needs to be handled very carefully given known human reasoning biases here!

  You’re maybe falling into a similar trap in the AI welfare section (perhaps conflating LM and Assistant and Assistant-instance beliefs?) but it’s pretty tricky territory!

• Oliver Sourbut 23 Feb 2026 16:08 UTC

  3 points

  0

  in reply to: Nathan Young’s comment on: Nathan Young’s Shortform

  One naive quick response: it’s a coordination problem. Sticking your head above the parapet alone gets you shot. And you only collect a very small fraction of the public good. (Also, publicly genuflecting instead may get you favours.)

  Meanwhile, quite rightly, coordination between firms and representatives of firms is seen as a big no-no in most sectors in a good liberal democracy, and this is enforced somewhat. It’d be market-distorting!

  One of the main reasons to have a government at all is to do the coordinating in such cases, because markets can’t (be trusted to). Difficult when the government itself is the (ex hypothesi) problem.

  That’s where checks and balances, and ideally ‘the wisdom of the people’ is meant to step in. (How is that going?)

• Oliver Sourbut 20 Feb 2026 9:30 UTC

  4 points

  0

  in reply to: tailcalled’s comment on: Linch’s Shortform

  Let me check: you mean the grandchildren have the same ex ante expected height as the ex ante expected height of the children. Of course! (Just as the children have the same ex ante expected height as the parents’ ex ante expected height, which is now screened off by knowing their actual height.) But if you reset your expectations based on the observed children’s heights, you’ll still witness (on average) regression to the mean.

  (If not this, I’d appreciate an explainer!)

• Oliver Sourbut 19 Feb 2026 11:12 UTC

  6 points

  2

  on: Build­ing Tech­nol­ogy to Drive AI Governance

  Strongly endorse something like this central point: governance is often driven by cost and affordances, so the availability of observation and intervention mechanisms could be a deciding factor in governance intent. (This makes affordances doubly useful, multiplying with existing intent while also unlocking more intent/​will.)

  (Foresightful governance intent can also conjure more affordance into being by resourcing and encouraging—or even requiring—work on it.)

  You also maybe half-say this: costs can include much more than financial. Side-effects, collateral, etc. can also be major costs (especially in governance). This makes e.g. privacy-protecting innovation look appealing.

• Oliver Sourbut 19 Feb 2026 10:47 UTC

  3 points

  0

  on: Man­aged vs Un­man­aged Agency

  Very good. Some terminological thoughts:

  ‘subagent’ is quite conflated. Sometimes we’re talking about the constituent parts of an emergent group actor. Sometimes we’re talking about a delegated (perhaps created) process. Sometimes we’re talking about different threads/​personas within a constructed multi-persona reasoning scheme. There might be other cases.

  ‘agent’ is also quite conflated. Sometimes we mean an actor in a principal-agent relationship. Sometimes we mean an AI. Sometimes we mean just someone/​something acting somewhat consequentialist. Sometimes we’re talking about something closer to a naked utility function.

  ‘shard’ is ehhh. A neologism that doesn’t seem to have a fixed meaning.

  I’m somewhat thinking aloud (but have given some thought to this previously). I’d tentatively suggest using terms like:

  • ‘delegate’ for a system which is either spun up whole cloth or conditioned to go do a particular thing

    • you can qualify if wanted, like ‘delegate agent’ (which implies a fair bit of autonomy) or ‘delegate process’ (which might be e.g. a relatively more mechanical sim or something)

  • ‘subprocess’, if needed, for something that really is more like a local, non-actuating search, computation, etc.

  • ‘principal’ for the actor doing the delegating

  • (personally I want to reserve ‘subagent’ for the thing which is a constituent part of something which is reasonably thought of as a group actor, but I don’t know if that use is sufficiently Schelling to win, and expect to need different words for that)

  • only sometimes ‘agent’, but often it’s worth tabooing that

• Oliver Sourbut 16 Feb 2026 14:47 UTC

  8 points

  5

  on: Align­ing to Virtues

  Broadly agree that entraining virtues is an encouraging path, especially for agentic systems, but also for systems used as building blocks in other beneficial tech.

  I have a terminological nit: ‘aligning to’ virtue feels like a type mismatch, to me. My conception of alignment (and I think this is moderately widely held, but am interested to hear if not) is that it’s centrally about objectives, preferences, goals, etc. (and one can even literally think of it as a geometric alignment in abstract preference-space if desired).

  So to me, ‘alignment’ carries a connotation of necessarily being about preferences. I might say

  • aligning (or alignment of) objectives

  • maintaining or enforcing deontological constraints

  • entraining (or embodiment of?) virtue

• Oliver Sourbut 15 Feb 2026 10:00 UTC

  2 points

  0

  in reply to: Oliver Sourbut’s comment on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  Confused by (strong?) downvotes on this. What’s the sin? If you disagree, use the disagree buttons (and I’d love it if you explain why you disagree too)!

• Oliver Sourbut 13 Feb 2026 11:09 UTC

  13 points

  6

  on: What We Learned from Briefing 140+ Law­mak­ers on the Threat from AI

  What is disarming is the absence of performance: clear, careful speech grounded in knowledge, and an evident commitment to honesty to oneself and others.

  The interesting thing is that it seems like this should be somewhat fakeable, but in practice it seems to be quite hard for people to fake. I have found this (being earnest, honest, and expert, not being fake!) to be my weird charisma superpower that I didn’t expect. I have also found that I don’t need to hide my passion and concern once a bit of human-to-human relating has happened over food or drink.
  
  Plus 1 to all your points Leticia, keep up the good work!

  As I recently said, I sense there’s a shift now where the demand for understanding (and battle for attention) remains, but serious people are now also demanding proposals and solutions, as you’ve noted. I suspect a failure mode here could be coming from the ivory tower with talking points without being responsive to the constraints and skills and options of the lawmakers (etc.) on the receiving end. That too will need to be a dialogue, perhaps a ‘tennis’ of sorts.

• Oliver Sourbut 12 Feb 2026 21:05 UTC

  37 points

  32

  on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  I realise that ‘person-affecting’ is signposted here, but (especially as someone who considers such views to be tremendously mistaken, but generally as someone who values clear discourse), I really think it ought to be emphasised and clarified more.

• Oliver Sourbut 12 Feb 2026 20:49 UTC

  9 points

  −2

  on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  I’ve only read the abstract, intro, and conclusion, and look forward to reading in more depth later.

  I wanted to criticise the central framing which seems implicit: either we build superintelligence (which can immediately solve near-arbitrary problems in medicine, governance, and other tech), or we don’t (in which case we have no hope of becoming wiser, more capable, or better coordinated).

  This strikes me as a terribly false dichotomy, very pervasive. We can totally get really ambitious benefits from tech, including those involving AI, without the autonomy and unified generality usually connoted by ‘AGI’ and ‘superintelligence’! And there’s a virtuous cycle where each boost to our wisdom marginally reduces the risks of badly-judged gambles.

  See AI for Human Reasoning for Rationalists for a brief expansion of this (and links to more).

  Perhaps there are unconquerable obstacles (which would nonetheless be conquered by ASI) which I don’t see clearly enough. I would like to know them (I fear the proliferation of destructive means via AI, and perhaps only a sufficiently surpassing singleton, and only one generated via superintelligence, sovereign or otherwise, can overcome this).

  Or perhaps this highly distributed and human-centred approach actually counts as (some form of) superintelligence on some views—that seems reasonable on first principles, though I think colloquially confusing.

• Oliver Sourbut 12 Feb 2026 10:26 UTC

  2 points

  0

  on: The na­ture of LLM al­gorith­mic progress

  Possibly better to have a fresh new post, and point each post to the other, rather than updating in place? (You could still keep the changelog, which is polite and helpful for your readers.) In particular, I only saw this revision by luck.

Strate­gic aware­ness tools: de­sign sketches

• Oliver Sourbut 9 Feb 2026 20:13 UTC

  3 points

  0

  in reply to: rvnnt’s comment on: De­sign sketches for a more sen­si­ble world

  Yeah, I like that. For me, it comes under coordination tech, which admittedly is a particularly thorny part of the map, and difficult. Ability to network and form coalitions, deliberate on models and prospects, and implement or commit to changes come under coordination, for me. Often there’d be an aspect of institution-installment or -adaptation, which deserves special care (forthcoming ‘charter tech’ discussion, name tbc). You’d also naturally want (individual foresight and collective) epistemics to be improved as an important support to that.

De­sign sketches for a more sen­si­ble world

De­sign sketches for an­gels-on-the-shoulder

• Oliver Sourbut 6 Feb 2026 17:05 UTC

  2 points

  0

  in reply to: Casey Barkan’s comment on: Another Cost Disease? We are all cap­i­tal­ists now

  Well, why do we envy? Evopsych just-so story says that, of course, others having much impinges on me whether I want it to or not (my security/​precarity, liberty, relative success in interactive situations, …).

  I think you can express that in a ‘goods’ framing by glossing it as the consumption of ‘status and interactive goods’ or something like that (but noting that these are positional and contingent on the wider playing field, which the bog standard welfare theorem utility functions aren’t).

• Oliver Sourbut 6 Feb 2026 2:14 UTC

  5 points

  0

  in reply to: Steven Byrnes’s comment on: The na­ture of LLM al­gorith­mic progress

  This might be some sort of crux between you and him, where he might point to growth in researcher focus/​headcount as well as perhaps some uplift from AI by now. (I might endorse the first of those a bit, and probably not the second, thought it’s conceptually reasonable and not out of the question.)