Oliver Sourbut

• Oliver Sourbut 19 Mar 2026 15:51 UTC

  2 points

  0

  on: Compradorization

  I mostly liked this. It’s making a few points, maybe roughly:

  1. GDP and ‘growth’

     1. GDP measures asset selloff and production of new durable means (capital) as the same; they aren’t

     2. asset selloff without converting that into new durable means implies (longrun) stasis at best, and more likely atrophy

     3. (so GDP ‘growth’ masks the actually longrun meaningful determinants, and can be misleading)

  2. institutions as determinants of trajectory

     1. ‘good’ institutions adhere to constituents’ interests, and convert windfalls into durable productive means

     2. ‘bad’ ones just extract windfalls, and maybe even hasten atrophy if they can make a buck doing so

     3. (this tends to determine whether the GDP ‘growth’ turns out to have been illusory)

  There’s also some helpful and opinionated elucidation of some terms of art like ‘Dutch disease’, ‘resource curse’, and ‘comprador’. I’d have thoroughly appreciated that essay. (I might have also liked to hear some prescriptions for alternative types of measurement and instrumentation that would reliably distinguish cases.)

  Then there are the psychological threads:

  1. Lacan and his unhinged menagerie

     1. ‘Perverts’ know what’s up but use distracting ontologies to mask it while making a killing e.g. comprador-style

     2. ‘Hysterics’ notice something’s up but misdiagnose and are deceiving themselves and others in some sort of self-aggrandising (?) dance?

        • Am I to understand these as something like controlled opposition? The perverts are erecting and nourishing these guys without them knowing?

        • Or somehow structurally selected and amplified by a larger malign multi-actor agent-agnostic process?

     3. ‘Neurotics’ fully buy the masking ontology and are selected to administer and enforce it

  2. An unspoken conspiracy

     1. Ultimately these structures perpetuate themselves

     2. This includes by attacking activities which operate outside of their ontologies (which would otherwise provide demonstration of the ontologies’ inappropriateness, like the Wickard v Filburn case)

     3. (I got the vibe that these are supposed to be everywhere, but I’m not sure if that’s explicitly stated)

  These parts felt maybe ‘fun’ and ‘poetic’ but on the whole a bit of a reach and harder to take seriously, while making the essay maybe twice as long as otherwise (or more). (I suspect this may be a taste thing and other readers may have loved that!)

  Maybe I feel a bit that these were expressed with more confidence or generality than I currently feel able to justify on the presented context (I do of course buy that these character types and dynamics can and do happen). I’m interested to know if my light summary above is on the right track, and whether there are core resources (I’d guess not Lacan himself) which would give justification to more of that, or would help diagnose when these things are more or less in operation. Similarly to the above bits, I also hungered for something like a tentative prescription or case studies of successful anti-menagerie-conspiracy outcomes.

• Oliver Sourbut 16 Mar 2026 8:59 UTC

  5 points

  1

  in reply to: XelaP’s comment on: Some Rules for an Alge­bra of Bayes Nets

  Yes, except the point was to ensure a DAG for any given sparse graph, not just any old (in this case a maximally dense) graph on those variables. As it happens, it’s indeed trivial or perhaps axiomatic to Bayes nets that there’s a DAG.

• Oliver Sourbut 13 Mar 2026 11:19 UTC

  19 points

  5

  in reply to: Elizabeth’s comment on: Eliz­a­beth’s Shortform

  Hmm. One of the most emotionally (?) difficult parts of deciding to leave the govt for me was knowing I’d give up my security clearance even though I’d already inferred or known ahead of time every supposedly ‘secret’ thing I’d been shown! (My clearance was not especially high level.)

  This was a new kind of flinch to experience, but I didn’t notice or reflect on the psychological generalisations and neighbours it might have at the time. Thanks for the nudge.

• Oliver Sourbut 11 Mar 2026 13:39 UTC

  2 points

  0

  on: Eco­nomic effi­ciency of­ten un­der­mines so­ciopoli­ti­cal autonomy

  I saw few mainstream economists take them into account when evaluating Trump’s tariffs

  Oh, my impression (not very studied!) was that nearly the whole conversation was about the tradeoff between efficiency (and who pays) and (re)onshoring. (The other part being the dynamic use as tactical economic weaponry.)

• Oliver Sourbut 11 Mar 2026 13:01 UTC

  2 points

  −2

  in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform

  nit: Christiano operationalised ‘slow takeoff’ via ‘world product’, not GDP. I’m not sure exactly what he meant by that (or if he had a more concrete operationalisation), but it does strike me as wise to not anchor to GDP which is awfully fraught and misleadingly conservative.

  ETA: fake news! I checked and while he starts talking about ‘output’, he later seems to operationalise it as GDP specifically

• Oliver Sourbut 4 Mar 2026 17:04 UTC

  2 points

  0

  on: I’m con­fused by the change in the METR trend

  My rough model of what’s going on (not published) is that ‘relevant effective evidence’ is what’s needed to succeed at a given subtask (of the cognitive kind that AI agents are being tested on here).

  Relevant effective evidence is accumulated in pretraining (data is information is evidence!) as well as through in-context evidence-gathering activities ^[1] .

  Generic webdata and the earlier corpora had some applicable data for these sorts of tasks. More recently, a greater fraction of the curated pretraining and posttraining data are dedicated to software engineering.

  (It’s complicated by changes to posttraining, but certainly ‘effective training data’ were increasing roughly exponentially with training date for some time, which maps to the exponential ‘time horizon’ via subtask completion chance.)

  This is all kind of annoying retrodiction but it adds up, at least in hindsight.

  1. ↩︎

     In-context evidence-gathering (i.e. exploration) competence comes from particularly generalisable learned heuristics, namely how best to experiment or try things out and interpret findings, but is still largely domain-specific.

• Oliver Sourbut 26 Feb 2026 9:10 UTC

  6 points

  0

  in reply to: Raemon’s comment on: Open sourc­ing a browser ex­ten­sion that shows when peo­ple are wrong on the internet

  If you want galaxy brain, check out my and Ben’s take :L (I’m heartened to see a fair bit of (independent?) convergence!)

• Oliver Sourbut 24 Feb 2026 12:13 UTC

  6 points

  2

  in reply to: Jan_Kulveit’s comment on: The per­sona se­lec­tion model

  Seems more than 2 years perhaps? (It’s (early) 2026!) Also I wouldn’t call this topically (though perhaps de facto socially ^[1] ) cyborgism.

  1. ↩︎

     Janus being among the earliest to write about these topics in detail

• Oliver Sourbut 24 Feb 2026 11:29 UTC

  4 points

  2

  on: The per­sona se­lec­tion model

  “What would the Assistant do?” (according to the beliefs of the post-trained LLM simulating the Assistant).

  nit: I think you’re slightly falling foul of the anthropomorphism you’re aiming to avoid, here. Rather, ‘according to the post-trained LLMs’s posterior over Assistant personas (further conditioned by the preceding context)’.

  More generally, a license to consider some things somewhat anthropomorphically (i.e. Assistant personas under certain training regimes) needs to be handled very carefully given known human reasoning biases here!

  You’re maybe falling into a similar trap in the AI welfare section (perhaps conflating LM and Assistant and Assistant-instance beliefs?) but it’s pretty tricky territory!

• Oliver Sourbut 23 Feb 2026 16:08 UTC

  3 points

  0

  in reply to: Nathan Young’s comment on: Nathan Young’s Shortform

  One naive quick response: it’s a coordination problem. Sticking your head above the parapet alone gets you shot. And you only collect a very small fraction of the public good. (Also, publicly genuflecting instead may get you favours.)

  Meanwhile, quite rightly, coordination between firms and representatives of firms is seen as a big no-no in most sectors in a good liberal democracy, and this is enforced somewhat. It’d be market-distorting!

  One of the main reasons to have a government at all is to do the coordinating in such cases, because markets can’t (be trusted to). Difficult when the government itself is the (ex hypothesi) problem.

  That’s where checks and balances, and ideally ‘the wisdom of the people’ is meant to step in. (How is that going?)

• Oliver Sourbut 20 Feb 2026 9:30 UTC

  4 points

  0

  in reply to: tailcalled’s comment on: Linch’s Shortform

  Let me check: you mean the grandchildren have the same ex ante expected height as the ex ante expected height of the children. Of course! (Just as the children have the same ex ante expected height as the parents’ ex ante expected height, which is now screened off by knowing their actual height.) But if you reset your expectations based on the observed children’s heights, you’ll still witness (on average) regression to the mean.

  (If not this, I’d appreciate an explainer!)

• Oliver Sourbut 19 Feb 2026 11:12 UTC

  7 points

  2

  on: Build­ing Tech­nol­ogy to Drive AI Governance

  Strongly endorse something like this central point: governance is often driven by cost and affordances, so the availability of observation and intervention mechanisms could be a deciding factor in governance intent. (This makes affordances doubly useful, multiplying with existing intent while also unlocking more intent/​will.)

  (Foresightful governance intent can also conjure more affordance into being by resourcing and encouraging—or even requiring—work on it.)

  You also maybe half-say this: costs can include much more than financial. Side-effects, collateral, etc. can also be major costs (especially in governance). This makes e.g. privacy-protecting innovation look appealing.

• Oliver Sourbut 19 Feb 2026 10:47 UTC

  3 points

  0

  on: Man­aged vs Un­man­aged Agency

  Very good. Some terminological thoughts:

  ‘subagent’ is quite conflated. Sometimes we’re talking about the constituent parts of an emergent group actor. Sometimes we’re talking about a delegated (perhaps created) process. Sometimes we’re talking about different threads/​personas within a constructed multi-persona reasoning scheme. There might be other cases.

  ‘agent’ is also quite conflated. Sometimes we mean an actor in a principal-agent relationship. Sometimes we mean an AI. Sometimes we mean just someone/​something acting somewhat consequentialist. Sometimes we’re talking about something closer to a naked utility function.

  ‘shard’ is ehhh. A neologism that doesn’t seem to have a fixed meaning.

  I’m somewhat thinking aloud (but have given some thought to this previously). I’d tentatively suggest using terms like:

  • ‘delegate’ for a system which is either spun up whole cloth or conditioned to go do a particular thing

    • you can qualify if wanted, like ‘delegate agent’ (which implies a fair bit of autonomy) or ‘delegate process’ (which might be e.g. a relatively more mechanical sim or something)

  • ‘subprocess’, if needed, for something that really is more like a local, non-actuating search, computation, etc.

  • ‘principal’ for the actor doing the delegating

  • (personally I want to reserve ‘subagent’ for the thing which is a constituent part of something which is reasonably thought of as a group actor, but I don’t know if that use is sufficiently Schelling to win, and expect to need different words for that)

  • only sometimes ‘agent’, but often it’s worth tabooing that

• Oliver Sourbut 16 Feb 2026 14:47 UTC

  8 points

  5

  on: Align­ing to Virtues

  Broadly agree that entraining virtues is an encouraging path, especially for agentic systems, but also for systems used as building blocks in other beneficial tech.

  I have a terminological nit: ‘aligning to’ virtue feels like a type mismatch, to me. My conception of alignment (and I think this is moderately widely held, but am interested to hear if not) is that it’s centrally about objectives, preferences, goals, etc. (and one can even literally think of it as a geometric alignment in abstract preference-space if desired).

  So to me, ‘alignment’ carries a connotation of necessarily being about preferences. I might say

  • aligning (or alignment of) objectives

  • maintaining or enforcing deontological constraints

  • entraining (or embodiment of?) virtue

• Oliver Sourbut 15 Feb 2026 10:00 UTC

  2 points

  0

  in reply to: Oliver Sourbut’s comment on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  Confused by (strong?) downvotes on this. What’s the sin? If you disagree, use the disagree buttons (and I’d love it if you explain why you disagree too)!

• Oliver Sourbut 13 Feb 2026 11:09 UTC

  13 points

  6

  on: What We Learned from Briefing 140+ Law­mak­ers on the Threat from AI

  What is disarming is the absence of performance: clear, careful speech grounded in knowledge, and an evident commitment to honesty to oneself and others.

  The interesting thing is that it seems like this should be somewhat fakeable, but in practice it seems to be quite hard for people to fake. I have found this (being earnest, honest, and expert, not being fake!) to be my weird charisma superpower that I didn’t expect. I have also found that I don’t need to hide my passion and concern once a bit of human-to-human relating has happened over food or drink.
  
  Plus 1 to all your points Leticia, keep up the good work!

  As I recently said, I sense there’s a shift now where the demand for understanding (and battle for attention) remains, but serious people are now also demanding proposals and solutions, as you’ve noted. I suspect a failure mode here could be coming from the ivory tower with talking points without being responsive to the constraints and skills and options of the lawmakers (etc.) on the receiving end. That too will need to be a dialogue, perhaps a ‘tennis’ of sorts.

• Oliver Sourbut 12 Feb 2026 21:05 UTC

  37 points

  32

  on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  I realise that ‘person-affecting’ is signposted here, but (especially as someone who considers such views to be tremendously mistaken, but generally as someone who values clear discourse), I really think it ought to be emphasised and clarified more.

• Oliver Sourbut 12 Feb 2026 20:49 UTC

  9 points

  −2

  on: Op­ti­mal Timing for Su­per­in­tel­li­gence: Mun­dane Con­sid­er­a­tions for Ex­ist­ing People

  I’ve only read the abstract, intro, and conclusion, and look forward to reading in more depth later.

  I wanted to criticise the central framing which seems implicit: either we build superintelligence (which can immediately solve near-arbitrary problems in medicine, governance, and other tech), or we don’t (in which case we have no hope of becoming wiser, more capable, or better coordinated).

  This strikes me as a terribly false dichotomy, very pervasive. We can totally get really ambitious benefits from tech, including those involving AI, without the autonomy and unified generality usually connoted by ‘AGI’ and ‘superintelligence’! And there’s a virtuous cycle where each boost to our wisdom marginally reduces the risks of badly-judged gambles.

  See AI for Human Reasoning for Rationalists for a brief expansion of this (and links to more).

  Perhaps there are unconquerable obstacles (which would nonetheless be conquered by ASI) which I don’t see clearly enough. I would like to know them (I fear the proliferation of destructive means via AI, and perhaps only a sufficiently surpassing singleton, and only one generated via superintelligence, sovereign or otherwise, can overcome this).

  Or perhaps this highly distributed and human-centred approach actually counts as (some form of) superintelligence on some views—that seems reasonable on first principles, though I think colloquially confusing.

• Oliver Sourbut 12 Feb 2026 10:26 UTC

  2 points

  0

  on: The na­ture of LLM al­gorith­mic progress

  Possibly better to have a fresh new post, and point each post to the other, rather than updating in place? (You could still keep the changelog, which is polite and helpful for your readers.) In particular, I only saw this revision by luck.

Strate­gic aware­ness tools: de­sign sketches