Oliver Sourbut

• Oliver Sourbut 18 Jun 2026 6:56 UTC

  9 points

  5

  in reply to: NunoSempere’s comment on: NunoSem­pere’s Shortform

  Might be understating Jamie’s rationalistadjacentness—take a look at societylibrary.org! I also met (and championed) Jamie through the FLF fellowship last year.

• Oliver Sourbut 13 Jun 2026 5:57 UTC

  3 points

  0

  in reply to: Bruce Lewis’s comment on: Ci­ta­tions Needed: Magic En­cy­clo­pe­dias to Save the World

  This is probably the biggest concern. I’m not centrally expecting universal uptake. Base rates aren’t great, though nonzero (including recently: Wikipedia, stackoverflow, elicit, reddit kinda, chatgpt kinda). There are a thousand ignored tools for every popular one. But there are truly new software potentials right now, and really great stuff could see wide uptake.

  Failing very widespread uptake, targeted distribution among those who have existing demand and discernment for well-grounded, full-picture background research is a satisfactory fallback (and perhaps beachhead).

Ci­ta­tions Needed: Magic En­cy­clo­pe­dias to Save the World

• Oliver Sourbut 11 Jun 2026 12:19 UTC

  3 points

  0

  in reply to: vigji’s comment on: Re­in­force­ment learn­ing scal­ing might in­cen­tivise hid­den rea­son­ing ar­chi­tec­tures for AI

  I’m not particularly well read there. I think there might have been some discussion in (and following) the model cards of the more recent Claudes and GPTs.

• Oliver Sourbut 7 Jun 2026 18:24 UTC

  11 points

  5

  on: Why Soft­ware Au­toma­tion Is Hard

  I’ve been pointing at a distinction between ‘enterprise’ software and ‘single-use’ software (usually from scratch, maybe single-user script, certainly single-‘author’ concept, …). It’s not entirely crisp and there’s clearly something like a continuum. This post is clearly describing ‘enterprise’ contexts (long-running or many-running, many-author, different stakeholders, integrated with various business concerns, …).

  Curious if others have good terms or characterisations of a distinction like this.

  Clearly AI accelerates single-use substantially, which meaningfully moves the waterline on what people even bother to try. (By the way I think one of the main effects may be an IoT and smart-device rennaisance!)

  Clearly ‘enterprise’ is a lot harder to invade—though one thing I think this post neglects is that it might reduce junior headcount (if it isn’t already), which has various effects. And it probably raises the waterline on passable and adaptable UX right away.

• Oliver Sourbut 5 Jun 2026 22:07 UTC

  2 points

  0

  in reply to: Bruce Lewis’s comment on: Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition

  Thanks! Yes, Elicit is (more than) on our radar—in fact Andreas gave a little feedback on this competition.

• Oliver Sourbut 5 Jun 2026 7:12 UTC

  3 points

  0

  in reply to: TsviBT’s comment on: Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition

  Brill! Please consider filling the interest form (and feel free to note that time constraint). For now that’s the deadline we’re looking at, but we’re fairly unsure what the spread of results will be and we have appetite beyond this timeframe and budget for working on these kinds of tools.

• Oliver Sourbut 5 Jun 2026 6:37 UTC

  2 points

  0

  in reply to: Chastity Ruth’s comment on: China won’t win the AI race but would it be much worse if it did?

  Do you think that remark is where the downvotes are coming from? :L

  Authoritarianism is mostly a domestic question rather than foreign so I don’t know if that has much bearing. (Though I’d note that foreign policy has also experienced what looks like some movement recently.)

  The articles linked in the original comment are three of mine (plus one Drexler). I’d say I’m concerned about all of catastrophic misuse, power entrenchment, gradual disempowerment, and (at some point, probably after we face some amount of those challenges) acute disempowerment/​extinction.

• Oliver Sourbut 4 Jun 2026 17:59 UTC

  2 points

  0

  in reply to: Jasper Blank’s comment on: Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition

  Certainly! Commenting here is a good start as were treating this crosspost as the first ‘forum’ (with comments) we link to. So others will hopefully see your comment here.

  We also have an expression of interest form and anticipate connecting people proactively (perhaps setting up a participants and spectators chat context).

• Oliver Sourbut 4 Jun 2026 17:56 UTC

  4 points

  0

  on: When AI Builds It­self (An­thropic In­sti­tute Linkpost)

  As I mentioned last week I’d love it if they’d acknowledge the potential for concentrated influence/​power risk in principle, even if they think they won’t or can’t do that.

Lab Leaks, Black Holes, and Eggs: Epistemic Case Study Competition

• Oliver Sourbut 4 Jun 2026 9:49 UTC

  3 points

  0

  in reply to: Oliver Sourbut’s comment on: China won’t win the AI race but would it be much worse if it did?

  Re

  unless something unforeseen happens, China is not winning the AI race

  the weird thing is that current US regime has been relaxing chip controls which seem like the main constraint and leverage point.

  Separately, because I think industrial automation might be where most of the geostrategic wins from AI come from, I actually think China has a more reasonable shot at that than you’re giving credit to. This ‘win’ looks less decisive and permanent to me than the imagined ‘superintelligence (or its wielder) takes over the world in short order’ but it might ultimately be where balance of relative global power comes from (which might affect future use of space resources and the way that subsequent tech and societal progress plays out).

• Oliver Sourbut 4 Jun 2026 9:25 UTC

  6 points

  1

  on: China won’t win the AI race but would it be much worse if it did?

  People are having different arguments because they weight various concerns differently (and sometimes also because they’re mind-killed).

  Many correctly recognise that the current Chinese state (and probably also the CCP regime) is mildly more authoritarian than the current US state. (Some note that the current US regime raises questions, but please don’t confuse the regime with the state, especially in a context where regime changes are frequent, as in the US.)

  Likely correctly, many also note that even contemporary, and certainly near-future AI, could be quite effective for concentrating and entrenching power, especially through scaled surveillance (mostly via automated analysis) and scaled coercion (e.g. maximally via weaponised drones and robots, more minimally through existing state policing, permissioning, credit, and so on).

  Some fear that China and other authoritarian states may go down this path. Some fear that the US and other somewhat authoritarian states may go down this path. (Either triggered by existing state/​regime or in a coup of some kind.) This could be bad because directly bad, and also bad because it’s probably not a good setting to encounter later technological transitions and/​or singularity from, which might come soon ^[1] .

  Separately, some people think that AGI and/​or superintelligence would yield a decisive strategic advantage (DSA) i.e. unassailable and uncounterable ability to project force and will (note that I think this is tremendously underexamined, and most people are utterly vibing when they talk about this). Or even short of that, just some large military advantage ^[2] that might give an imperialist power more confidence in grabbing territory, perhaps even world domination. If CCP does this, the argument goes, the world gets more authoritarian probably. (If another does this, does it get less?)

  Taking most of the above together, and choosing parameters and credences somewhat carefully, that can give some kind of (I think sloppy!) argument that, absent coordination, racing is justified. The entirely missing mood is that this racing is tremendously regretful and we should do what we can to coordinate out of it. If you rule out DSA but still think there’s gradual economic, military, and industrial advantage (I think this is something like ‘mainstream automation-pilled perspective’ held by many govt and investors) that looks like even more reason to race.

  Separately again, many people think that AGI and beyond will be very difficult to control, leading to either acute or gradual disempowerment/​destruction. That tends to point people toward the frame that racing is terrible!: it’s just competing to be the one with the privilege of killing everyone (including self). (Others still think that would be good actually, for misanthropic or other reasons.) This is maybe central LW perspective?

  Others think the cat is mostly out of the bag, motivating coordination for pacing, and unilateral adaptation and defence work, while aiming to get coordinated and secure enough to delay, prevent (or somehow survive?) a transition far past AGI.

  1. ↩︎

     There’s some case that could be made in favour of a more locked-down society in the run up to a singularity—I think some versions of the superalignment kind of strategy are implicitly imagining this, in fact; rarely spoken out lout.

  2. ↩︎

     My main candidate for this is plain old industrial capacity, with something like ‘better combined ISTAR’ as a close second. But your enemy can still MADly nuke you in those worlds.

• Oliver Sourbut 4 Jun 2026 8:54 UTC

  5 points

  0

  in reply to: Mo Putera’s comment on: Dis­solv­ing the Deep Learn­ing Sam­ple Effi­ciency Gap

  Uh, maybe. There’s some scattered stuff. I’m not sure if anyone else is thinking about it the way I do, but probably it’s some of the ‘hierarchical’ (keword) RL/​control and robotics people if anyone. e.g.

  • Planning in a hierarchy of abstraction spaces

  • Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning

  • Option-Critic

  • DeepMimic

  • Human-Level Reinforcement Learning through Theory-Based Modeling, Exploration, and Planning

  • Model-based hierarchical reinforcement learning and human action control

• Oliver Sourbut 4 Jun 2026 8:43 UTC

  3 points

  0

  in reply to: Steven Byrnes’s comment on: Dis­solv­ing the Deep Learn­ing Sam­ple Effi­ciency Gap

  Is it plausible to you that there’s some equivalent to ‘mental simulations’ humans use for model-based world-sample-efficient learning?

  If so is it plausible they are mostly (overwhelmingly?) subconscious? If so could those be much (orders of magnitude?) faster than equivalent real-world interaction?

  (I tentatively think yes to all of these, mostly because of the computer science of it, without much context on the neuroscience of it. cf my other comment on temporal abstraction.)

• Oliver Sourbut 3 Jun 2026 13:11 UTC

  10 points

  2

  in reply to: Linch’s comment on: Linch’s Shortform

  Did anyone anywhere notice your Borges reference?

• Oliver Sourbut 2 Jun 2026 10:12 UTC

  5 points

  1

  on: Dis­solv­ing the Deep Learn­ing Sam­ple Effi­ciency Gap

  RLHF and RLVR are model-free: they update the policy from sampled real trajectories, with no learned model of the environment to plan or imagine inside.

  It’s perhaps worth distinguishing the learned reward model of RLHF—so the policy isn’t updated only from ‘real’ trajectories but also from exploratory hypothetical trajectories scored against the learned reward model.

  In RLVR, one could claim that the whole point is that we have the ‘model’ - the training environment—and it’s arguably not only more accurate to just run that directly rather than imagining it, but also cheaper in many cases (e.g. it’s shell scripts and file manipulations, not a giant NN world model). Not always: some environment interactions might be very slow (e.g. running a big gradient descent). You write “AI labs have focused their RL work on areas where the environment and verification step are cheaper to compute directly rather than in a learned world model”, which acknowledges this.

  One thing you haven’t mentioned (which I think is quite crucial in human sample-efficient planning and learning) is temporal abstraction. That’s something that only some world models can do (ones which flexibly represent ‘options’ at various temporal granularities rather than merely representing atomic ‘action’ transitions). Humans definitely do this. I’m fairly confident that the implicit world models baked into large pretrained NNs also do this to some extent in their internal planning. I haven’t seen many architectures which allow such a temporally abstract world model to interface with policy (e.g. in actor-critic/​dreamer style) to allow improved sample-efficiency. EfficientZero, with its ‘value prefix’ model, comes closest (and not that close) among things I’ve paid attention to. Notably, language encodes all kinds of temporal abstraction essentially arbitrarily flexibly (this realisation was what prompted me onto the LLM-agent train back in 2021).

  Given working temporal abstraction, even RLVR on cheap envs might be substantially accelerated.

• Oliver Sourbut 1 Jun 2026 13:57 UTC

  3 points

  0

  on: Fi­nan­cial Costs of an AI Pause?

  I don’t pretend to be an economist but this superficially looks to conflate market with economy. I would guess this is bound to run into issues.

  Another quick note is that markets funge on very short timescales as growth investors move their money between the apparently-most-promising growth stocks (e.g. several former growth stocks have dropped alongside the AI stock growth). ^[1]

  The wider economy funges similarly, but on a longer timescale as financial investment moves into capital investment and human capital adjusts.

  The longer-term economic impact routes, presumably, mostly through the productivity impacts (i.e. broader technological and automation).

  1. ↩︎

     i.e. investment moving out of industry/​stock A doesn’t disappear but moves into its next-best industry/​stock B

• Oliver Sourbut 1 Jun 2026 13:33 UTC

  8 points

  1

  in reply to: leogao’s comment on: leogao’s Shortform

  Yeah, I guess I’m being provocative, but I actually do think that.

  • Survey (sampled, point-in-time, framing-dependent, out-of-context) is an (often quite good) analogy for any specific (or general) concept of what people believe.

  • Experiment is a good case: one kind of ‘really good’ evidence is obtained by driving out as many confounders, controlling as many variables as possible, then reasoning that ‘by this really really close analogy, the same or very very similar should obtain under what look like the same conditions’. Usually there are residual stochasticities. Often experiments are out of context in some way, or the conclusions we would like to draw are extrapolating some amount from what we actually checked.

  • ‘Read a study’ inherits the analogy-ness of whatever experiments it’s based on, plus I guess the thing where by analogy to those other times you read studies that seemed to be accurate (by the same person or same team or same org or same discipline or same culture or same species, …), you expect it to be accurate.

  • Mathematical proof, as always, is a slightly annoying limit case. I could probably say something like, ‘well, any conclusion other than the mere mathematical conclusion is smuggling in some analogy (to quantities, types, relations, whatever)’, and perhaps also your mathematical methods are trusted by analogy to… errr the other times logic helped you, but I don’t know to what extent I’d really stand by that.

• Oliver Sourbut 1 Jun 2026 9:45 UTC

  4 points

  2

  in reply to: Jonas Hallgren’s comment on: The main im­pact from au­to­mated AI pro­duc­tion: con­cen­tra­tion of power?

  I agree something like the ‘dynamics of the firm’ (cf Coase) might be in for a shakeup: maybe that means many more small orgs (Coase might say: reduced transaction, marketing, discovery, and contracting costs, …); maybe it means aggregation and concentration (Coase might say: more expansive and responsive holistic/​top-down context management, increasing (or less decreasing) returns to scale, fewer management mistakes and follies, …).

  With this post I’ve leant into the hypothetical of ‘fully automated AI R&D soon’, and considered the context of frontier AI companies. I don’t feel confident giving any bold statements about wider economic effects, only that it’s likely to shake certain things up!