Vladimir_Nesov

Karma: 37,528

Vladimir_Nesov 28 Mar 2026 16:28 UTC
3 points
0
in reply to: Mateusz Bagiński’s comment on: Roman Malov’s Shortform
If it’s the next level of pretraining compared to Opus 4 and Gemini 3 Pro, there’s potential for novel observations about what that does to the texture of capabilities. It’s the kind of thing that will predictably scale further soon without requiring algorithmic breakthroughs, and it’s not even clear that RLVR can be expected to deliver more phase changes in capabilities in the near future due to pure scaling than pretraining (even if it’s less than 1 phase change for either in expectation, until 2032 or so).

Vladimir_Nesov 28 Mar 2026 16:20 UTC
3 points
1
in reply to: Roman Malov’s comment on: Roman Malov’s Shortform
Introducing three notable classes of model sizes (Sonnet, Opus, above-Opus) is possibly the consequence of Anthropic needing to feed datacenters with three different classes of servers during the Claude 5 lifecycle: the smaller Nvidia 8-chip servers (H100/H200/B200), rack-scale Trainium 2, and TPUv7, each being able to serve larger models than the previous one efficiently. Meanwhile, OpenAI until very recently was stuck with mostly the 8-chip Nvidia servers and so had to use smaller models (they couldn’t serve their own Opus-class model efficiently), and only now they’re getting enough GB200/GB300 Oberon racks to offer an Opus-class flagship model soon. Though the Blackwell Oberon racks are better than Trainium 2, so there’s some advantage to what OpenAI will be able to serve compared to Opus 4, all else equal. And based on GPT-5.4 (which is likely in Sonnet’s weight class), currently OpenAI might be better at RLVRing capabilities than Anthropic for models of the same size, so OpenAI’s new Opus-class model might end up notably better than Opus 4. But by that time or a bit later Opus 5 will be released, so even if these considerations are on point, it’s still unclear which of them wins in the Opus weight class during late 2026.

Based on hardware considerations, I expect the prices per token for the above-Opus class of Anthropic model will start out high, maybe 4x the price of Opus 5 (which probably won’t change much compared to Opus 4), because they’d need to serve it on suboptimal hardware initially. And then the prices go down to maybe 2x the price of Opus 5 at the end of the year once the TPUv7 datacenters go online. This is what happened with Opus 4 over 2025, as Trainium 2 datacenters came online later in the year.

Vladimir_Nesov 26 Mar 2026 22:45 UTC
4 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform

Like a method you have available as a human is learning on the task itself and AIs also have this option, but due to garbage sample efficiency it wouldn’t work.

Sample efficiency for RLVR is plausibly good enough if relevant tasks and RL environments can be automatically formulated, which is currently out of reach but it’s unclear for how long it stays this way. And for in-context learning, sample efficiency could get higher if it worked well enough (as learned learning, it’s in principle not constrained by what hardcoded learning can do), it just doesn’t work well with pretraining.

Vladimir_Nesov 26 Mar 2026 20:55 UTC
2 points
0
on: What if superintelligence is just weak?
Dangerous capabilities well short of superintelligence are followed by overwhelmingly catastrophic capabilities 20 years later. But superintelligence being impossible, or its sudden emergence (in a matter of years) being impossible, is a position that makes whatever happens 20 years after a more modest milestone less relevant, because whatever happens a few years down the line (such as the state of alignment and control) is shaped by what’s done before that, and there’s time to figure things out.

Gradual disempowerment is a relevant argument within that framing. Disputing the framing involves arguments that sudden superintelligence is possible, or that eventual superintelligence is a phase change that preceding work won’t prepare for (rather than an arbitrary point in a gradual process not distinct from all other points). Disputing the framing is more difficult, but accepting this framing makes people much more tolerant of continuing unbounded development of increasingly capable AI at the pace that the technology itself is asking for. So these two models of AI danger are not very aligned on policy.

Vladimir_Nesov 26 Mar 2026 14:47 UTC
9 points
0
on: Who’s Afraid of Acausal Trades?
Acausal trade is not about trade (or pursuit of particular goals), it’s about expanding the scope of coordination. When things in multiple places, or at multiple times, or in multiple possibilities can jointly decide what to do, with each instance carrying out its part according to a shared policy, that’s coordination.

A given agent/person is mostly coordinated across the instances, at different points in time, and between different possibilities (different possible situations that might be mutually exclusive in one timeline or not). Exploiting existing coordination is best done with updateless policy selection, to the extent that’s possible. But this doesn’t help with establishing coordination in the first place, especially across multiple agents that originally didn’t think about each other (but could benefit from acting in concert). Logical updating (starting to listen to some computation that would influence a policy, perhaps a contract shared with other parties, which likely breaks some properties that updatelessness wants) has the character of enabling new coordination to be established, and the idea of acausal trade is gesturing at this, the kind of thing that happens in the Prisoner’s Dilemma.

It seems strangely difficult to introduce pursuit of particular goals in protocols for establishing coordination that work this way. Possibly this is a clue that the process of carrying out logical updating that establishes coordination (once it’s decided how to coordinate/update) should be distinct from the process of deciding how to coordinate/update in a way that pursues particular goals.

This is probably not very legible, so here’s a more concrete sketch. A contract (taking the form of a computation that its signatories will let influence their policies or thinking) should be chosen according to one’s values, but not based on the specific consequences of adhering to it (which will typically remain unknown at least until-in-logical-time everyone signs, and then the contract needs to take a look at what it’s dealing with). Once signed (which can be an updateful step for the signatories in some sense), a contract that is itself an agent can (updatelessly) exploit coordination among the signing parties built in terms of its behavior across their situations, (updatelessly) pursuing its own values, which can be distinct from those of the signatories that listen to its policy in concert. This policy can depend on who decides to sign it, which is how an individual potential signatory gets to influence the contract’s behavior, which in turn influences the behavior of the other signatories that would listen to this shared contract.

Vladimir_Nesov 26 Mar 2026 5:41 UTC
21 points
7
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
Pretraining memorizes all the facts in the world, but only gives weak fluid intelligence (in-context learning). RLVR trains crystallized intelligence that expresses itself in-context as strong fluid intelligence (which isn’t fake or illusory, within its scope), but this narrow strength falls apart sufficiently out of distribution (regressing to pretraining levels). Thus jaggedness (in fluid intelligence) is a good way of framing this, and currently the jaggedness profile is determined by RLVR training, the topics that were sufficiently covered in the RL training data. Humans are different in having a higher baseline of general fluid intelligence (than what pretraining gives LLMs), and thus in often possessing fluid intelligence that’s stronger than crystallized intelligence for the same topic, while LLMs always have strong crystallized intelligence for the topics where they have strong fluid intelligence (those covered by RLVR training).

General “smartness” might significantly improve from replacing pretraining with something more effective, applying RLVR much more broadly, or figuring out how to automatically train in response to post-deployment data (bringing it in-distribution for RLVR levels of in-context learning capability). None of these things are currently assured to be on track to happen quickly. The most straightforward step in this direction is automation of routine AI R&D that helps with applying RLVR during training to many more topics than is humanly feasible, expanding the areas covered by strong fluid intelligence. But even that plausibly runs into a wall of jaggedness in what kinds of things can be trained with RLVR, and of the absence of post-deployment training (at RLVR levels of capability).

So I think it remains plausible that “powerful AI” (that fully automates civilization) isn’t near. But also the scope of topics where AI is smart will increase in the next few years, and the rising tide of pretraining scale will improve the fallback level of smartness outside these topics (or for sufficiently novel constructions within these topics). It remains to be seen if this is sufficient to overcome the jaggedness of RLVR, either by sufficient competence at gluing narrow capabilities together (at pretraining levels of fluid intelligence), or by automating self-application of RLVR by LLMs to themselves that fixes gaps in RLVR-level capabilities for novel topics, as soon as they come up. Failing that, more breakthroughs may be required, which could take a hard-to-predict amount of time, making 10-20 years to “powerful AI” possible despite the current speed of progress.

Vladimir_Nesov 24 Mar 2026 14:55 UTC
3 points
0
in reply to: Linch’s comment on: Linch’s Shortform
I also don’t see an option to publish to Alignment Forum (from an LW draft). There is a “Move to Alignment” button in the menu for a post on the list of Drafts (the menu is under the three-dots-vertically button that appears on mouse-over on the draft post line, but otherwise doesn’t show). I guess it could be published there this way, but there’s nothing there about the EA Forum. Though I’m not sure if my account is considered registered on the EA forum (if this happens on its own by default), not being registered there might be the reason the option doesn’t show for me.

(There was a change in the post interface recently, and these features might’ve been lost. Though presence of the “Move to Alignment” thing could be the reason to get rid of an explicit checkbox when posting, the alternative method feels unnecessarily hard to find, so plausibly not downstream of a decision to make it this way, more like a consequence of practical difficulties. I only found it because I knew the feature existed previously, otherwise I might’ve assumed the feature is absent. Maybe I’m just failing to notice this in the new interface, where it should be obvious, like perhaps after pressing the “Publish” button there is an additional page, but since I’m not actually publishing anything now I can’t safely check.)

Vladimir_Nesov 22 Mar 2026 21:55 UTC
9 points
0
in reply to: speck1447’s comment on: speck1447′s Shortform
Cognition could have multiple parts that have nothing to do with each other, yet evolution (which has no mind at all) still found them. Thus it’s difficult to rule out that scaling some general learning method might find more of such parts, even when they aren’t found at lower levels of compute. While rapid scaling continues, it’s hard to see if there are genuine obstructions that won’t fall to feasible scale.

An LLM with a chain of thought is a general-purpose computer. In principle it could be implementing any cognitive algorithms, if the learning process gets the weights to do that. So a claim that an LLM can’t do something is a claim that the learning methods won’t train it to do it. With more scale, learning methods work better. And using RLVR, it might be possible to teach LLMs to experiment with methods for training themselves to do more things (implementing more parts of cognition in the weights), much more efficiently than evolution did to discover the human brain.

Vladimir_Nesov 21 Mar 2026 4:59 UTC
2 points
0
on: AI #160: What Passes For a Pause

Nvidia to spend $26 billion to build open weight AI models. I am marking my Nvidia investments down to reflect a $26 billion lower target market cap. If Nvidia wants to invest in AI models they should invest in companies building AI models.

Nvidia is partnering with Mistral AI for at least some of this. Again, I would be investing instead, and not in Mistral.

Nvidia benefits from a broad ecosystem of smaller AI companies and API providers, in part so that a few giant AI companies won’t have too much negotiating power when buying Nvidia’s systems. Google has TPUs that are in some ways better, and AWS’s Trainium is a credible alternative (at least for internal use), so these companies already don’t need Nvidia as much as others do, and this could happen for the new giant AI companies as well at some point.

Vladimir_Nesov 18 Mar 2026 19:57 UTC
11 points
6
in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform
I’m starting to suspect that if 2026-2027 AGI happens through automation of routine AI R&D (automating acquisition of deep skills via RLVR), it doesn’t obviously accelerate ASI timelines all that much. Automated task and RL environment construction fixes some of the jaggedness, but LLMs are not currently particularly superhuman, and advancing their capabilities plausibly needs skills that aren’t easy for LLMs to automatically RLVR into themselves (as evidenced by humans not having made too much progress in RLVRing such skills).

This creates a strange future with broadly capable AGI that’s perhaps even somewhat capable of frontier AI R&D (not just routine AI R&D), but doesn’t accelerate further development beyond picking low-hanging algorithmic fruit unlocked by a given level of compute faster (months instead of years, but bounded by what the current compute makes straightforward). If this low-hanging algorithmic fruit doesn’t by itself lead to crucial breakthroughs, AGIs won’t turn broadly or wildly superhuman before there’s much more compute, or before a few years where human researchers would’ve made similar progress as these AGIs. And compute might remain gated by ASML EUV tools at 100-200 GW of new compute per year (3.5 tools occupied per GW of compute each year; maybe 250-300 EUV tools exist now, 50-100 will be produced per year, about 700 will exist in 2030).

Vladimir_Nesov 17 Mar 2026 22:06 UTC
35 points
0
on: Vladimir_Nesov’s Shortform
Eight-rack Oberon scale-up worlds for Rubin might be in the works, which potentially makes them ready for models with tens of trillions of parameters one year earlier than Kyber racks would’ve made this efficient with Rubin Ultra, in 2027-2028 rather than in 2028-2029.

There were some unclear communications from GTC 2026 about a two-layer all-to-all NVLink scale-up world called “NVL576”. This seems to be a system comprising 8 non-Ultra Rubin Oberon (as in NVL72) racks (each with 144 compute dies in 72 2-die packages), so 576 packages (1152 compute dies) across 8 racks. It’s confusingly announced as “Vera Rubin Ultra NVL576 will combine eight … racks, each with 72 Rubin Ultra GPUs”. (Another slight confusion when searching about it is that in 2025 “NVL576” referred to a single Rubin Ultra Kyber rack with 576 compute dies in 144 4-die packages, but that’s clearly a different system from the “NVL576″ announced at GTC 2026.)

An 8-rack Rubin Oberon NVL576 system would have 165 TB of HBM4, so inferencing (and RLVRing) models with tens of trillions of parameters won’t be significantly less efficient than for models with trillions of parameters. This was TPUv7′s advantage (full buildout in 2026), and last year Nvidia only announced plans to close the gap with the Kyber rack for Rubin Ultra (576 compute dies in 144 4-die packages in one rack, 147 TB of HBM4E), which is due to come out in 2027, so that full-scale buildout would only conclude in 2028 (maybe early 2029). But Vera Rubin Oberon systems are already in production, and full-scale buildout will happen in 2027 (maybe early 2028, for some larger datacenter sites getting fully online).

So if these two-layer scale-up worlds for Oberon are available from the start, the constraint of HBM per scale-up world gets lifted a year earlier, which might translate into models with tens of trillions of parameters getting RLVRed and becoming available a year earlier on non-TPU systems. This might be especially crucial for OpenAI (if models this large can be made more capable than 10x smaller models in the relevant timeframe), since they are mostly working with Nvidia hardware, but even for Anthropic this might make a difference (they are getting 1 GW of TPUv7 in 2026, but it’s unclear if they’ll be able to get meaningfully more TPUs in 2027-2028).

(Having enough hardware to efficiently serve inference for a model of some shape is necessary to deploy it as a flagship model. If instead most of the available hardware is only good at serving smaller models, then even if the larger model can be trained, it can’t be served to most of the users as cheaply as the better hardware allows. This makes it less likely that it gets trained in the first place, and so the capabilities of the most popular hardware indirectly translate into the shapes of models that get trained in practice, even when it’s possible to train larger models in principle, and these larger models could still be served a bit slower and more expensively on older hardware.)

Vladimir_Nesov 14 Mar 2026 23:33 UTC
14 points
0
in reply to: teradimich’s comment on: Sparks of RSI?
My timelines didn’t notably update overall (I wrote this comment before re-reading the comment you linked). Automation of routine AI R&D is an answer to the question of what specifically causes AGI/RSI in 2026-2027, if it happens this early, but in my model getting a clearer sense of what form this might take doesn’t make AGI in 2026-2027 more likely. Most of my AGI probability is in unknown breakthroughs or scaling outcomes (where quantity becomes quality in a capability that wouldn’t a priori obviously be able to go that far, before the necessary quantity actually arrives). These things are enabled by more compute and then either follow quickly (low-hanging fruit at a given level of compute, unlikely to be accessed earlier even if possible in principle) or take multiple years (when needing human-invented conceptual advancements). As compute grows faster/slower, this directly influences the probability of AGI/RSI per year during the few years after that.

I expect compute buildout (for individual AI companies) to continue at the current pace (of 2-4x more becoming available each year) in 2022-2029, perhaps with low-hanging fruit getting picked through 2032, then slower growth in 2029-2035, and even slower after that (absent AGI). Without AGI by 2035-2045, a lasting ban/pause gets more likely as global cultural attitudes might change. So the highest per-year probability is in 2027-2032, then notably lower in 2032-2038, and even lower after that. And I’m placing the median in 2032-2033. Which means 10% per year in 2027-2032, extending the first 10% to 2026-2027 since there’s some visibility into the very near future that says this probably isn’t happening right now.

New-for-me considerations from mid 2025 to now are a clearer picture of capabilities of RLVR and its implications for AI company revenues, and some details on what might happen around the Rubin Ultra buildout. Turns out RLVR works for IMO gold even with relatively small models (DeepSeek-V3), and there are now some LLM solutions to technical open problems, so it’s probably sufficient for training the deep skills aspect of AGI (it’s more than mere elicitation), especially with bigger models. Though jaggedness still makes it less useful than that suggests. This made 100 billion dollar revenues (2-5 GW training systems) for AI companies before 2030 more likely than o1/o3 suggested on their own. Scaffoldings like Claude Code, especially with better post-deployment adaptation (what’s being foreshadowed as “continual learning”), make even trillion dollar revenues before 2030-2032 plausible (which means 30-50 GW training systems, but it’ll take more time to scale the supply chains and actually build that with hardware of the same generation, as a single system for an individual AI company, probably closer to mid-2030s).

For the Rubin Ultra buildout (2028-2029), individual 5 GW systems don’t seem to be in the works, which previously seemed to suggest it already starts a slowdown in the trend, which then only lasts at the current pace during 2022-2026, and goes 2x slower in 2026-2029. (Trillion dollar revenues only extend the slower part of the trend, after the initial slowdown somewhere in 2026-2029, as the supply chains struggle to catch up to the available funding, and before compute mostly stops growing other than through improved price performance of hardware.) But Nvidia’s bet on FP8 in Rubin makes 2027-2029 hardware 2x-4x more performant per GW than I expected (2x from chips that are faster in FP8 because they no longer care about BF16 as much, and maybe another 2x from the confidence that FP8 is a first-class citizen in training of even the largest models, where this confidence wasn’t already priced in). So even 2 GW Rubin training systems remain on trend, even though the trend previously asked for 5 GW training systems for the same compute. The fact that Nvidia is making this bet means others will likely be doing the same, so this doesn’t necessarily only concern OpenAI.

Vladimir_Nesov 14 Mar 2026 19:29 UTC
6 points
2
in reply to: Nathan Helm-Burger’s comment on: Sparks of RSI?
There are sparks all over the place right now, sure. What matters is which of them get to contribute to wildfires, and which are lost in those wildfires started by completely different sparks, never given the opportinity to develop because of the order in which things happen. Not all sparks are sparks of wildfires. RSI is the wildfire, and not all sparks are relevant to RSI. Especially in actuality where something else happens first and ends their relevance, rather than in principle where in isolation and with enough resources they could be developed further.

Specificity is important when there are so many sparks. So I’m gesturing at a specific spark, automation of routine R&D, that plausibly might actually cause a wildfire before the other sparks grow similarly dangerous. Maybe it doesn’t catch, but in that case the other things still need to have a path towards learning of novel RLVR-level skills to have the potential, and many of them probably don’t, at least on their own.

Non-specific sparks matter for defense in depth, as in computer security where you harden the system against even the patently impossible interventions wherever it’s not too costly to do that. AI is existentially dangerous and nothing remotely close to the current methods can change that. But most sparks don’t matter for forecasting what is going to actually happen.

Vladimir_Nesov 14 Mar 2026 18:35 UTC
16 points
1
on: Sparks of RSI?
When RSI is not unbounded (doesn’t take steps that eventually proceed far beyond the modern civilization), it shouldn’t count as RSI. Some test-time training things might work better than pure in-context learning, but not all post-deployment improvement is RSI, and not all RSI must happen post-deployment.

The most near-term path to RSI that seems plausible to me is not about any sort of continual learning or scaffolding, but automation of routine AI R&D, because it gets to leverage RLVR, the only method of training a wide variety of deep skills that currently works with LLMs. Automating application of RLVR to LLMs (part of routine AI R&D) thus suggests the possibility of genuine RSI. On the other hand, all the post-deployment things don’t have the immediate potential to make LLMs able to play good chess, or to become fluent in a novel topic of math (that is, something not already trained into the LLM with RLVR before deployment). These things might help indirectly though, as part of automated routine AI R&D, and of training LLMs to get better at routine AI R&D.

Vladimir_Nesov 14 Mar 2026 17:53 UTC
7 points
0
in reply to: Linch’s comment on: Linch’s Shortform
Values need to be developed and decided, they aren’t already settled to merely be obeyed or pursued. If values are being developed not in accordance with the volition of their originators, they don’t have legitimacy. Originators of values must personally have a hand in developing them further in some way, and plausibly all they could do (in many possible situations) contributes to that to some extent. Thus values are in content closer to the world (or just a home, with more personal influence) you are living in and building (across many possibilities, to counteract path dependence), than to directives you must follow.

Vladimir_Nesov 14 Mar 2026 17:04 UTC
2 points
0
in reply to: Linch’s comment on: Linch’s Shortform

Superintelligences (likely digital intelligences, though in theory could also be our posthuman descendants)

If takeoff happens soon, either many of the currently living humans will at some point have an affordance to personally grow towards superintelligence, or no descendants of humanity will ever be in that position.

Vladimir_Nesov 11 Mar 2026 19:22 UTC
12 points
2
on: GPT-5.4 Is A Substantial Upgrade
I’m guessing GPT-5.2 uses a new pretrain, maybe pretraining distillation from the new big (GPT-4.5 class) model that still can’t be released or properly RLVRed due to insufficient compute in the form of NVL72 servers. And GPT-5.4 is a new level of RLVR scaling on top of that pretrain, where GPT-5.2 itself only aspired to mildly exceed GPT-5.1, which might be essentially o3 refined to be less of a lying liar, possibly still based on the same base model as GPT-4o.

With this level of performance for a smaller model, a big model that will be likely released later this year could be significantly stronger than the current best Anthropic and GDM models, though by that time Opus 5 might be out. And Gemini 4 could take advantage of TPUv7 to become by far the largest model yet, but that’s end of 2026, likely after OpenAI releases their big model, so there’s still a window where OpenAI plausibly takes the lead.

Vladimir_Nesov 10 Mar 2026 18:46 UTC
3 points
0
on: Contra Myself on Free Will
Bridging between physical computers and abstract programs seems helpful as a prerequisite to thinking about free will. (Similarly, between real physical systems in some regime such as classical mechanics, and corresponding abstract physical systems.) A physical thing determines its abstract model, letting it be observed, but also an abstract model can serve as a design for a physical thing that follows its structure. When an abstract model has its own internal dynamics, the corresponding physical thing would need to remain in step with that abstract dynamics.

The correspondence goes both ways: the abstract dynamics, as a design, must determine the physical dynamics, and also the observations of how the physical dynamics proceeds must be useful as observations of how the abstract dynamics proceeds. A physical calculator computes a certain answer because of what the abstract fact of arithmetic determines that answer to be, but also observing what the physical calculator computes lets us learn what the abstract fact of arithmetic says.

A physical human is confusing, because there are too many factors involved. But an abstract human that exists on their own, or better yet an abstract decision making deliberation, can be considered in isolation, as a process that proceeds exclusively according to its own abstract dynamics. The physical world then has no choice but to remain in step with that dynamics, or else the abstract dynamics is not a faithful model of what takes place in the physical world. Decisions taken by the abstract deliberation can’t let it escape the correspondence with the physical world. But also it’s free to reach any results that it does, and the physical world would need to comply with them, the same as a physical computer would need to comply with what the abstract program running on it is doing.

Vladimir_Nesov 10 Mar 2026 18:06 UTC
8 points
1
on: Vladimir_Nesov’s Shortform
If automated learning of deep skills is a crux for AGI, then automating AI R&D becomes more relevant in the sense of routine application of the current methods. Routine AI R&D is then a milestone on the way to AGI, while frontier AI R&D is merely an indicator that AGI has been achieved, not relevant to AGI timelines. This milestone plausibly doesn’t need any new breakthroughs or even significant scaling.

The distinction between the capability for carrying out tasks and the depth of skills that can be learned thus becomes the distinction between manual training of AIs by humans with routine application of current methods (the current situation) and its automation (the milestone where AIs apply those same methods on their own to train new skills into the next version of the AIs). Routine AI R&D is itself merely a collection of mostly fixed skills, doesn’t involve acquisition of new deep skills, and so probably can be trained into AIs by humans using routine AI R&D. In contrast, frontier AI R&D can require developing and mastering novel deep skills or theories, and so can’t be done automatically by an AI that is incapable of automatically learning deep skills.

It’s plausible this on its own is sufficient for AGI in the sense of the full automation of civilization. This is something that can be vaguely predicted with capability benchmarks (unlike the more direct attacks on remaining hobblings), and improves with inputs to capabilities such as the amount of compute.

Thus I think when people talk about imminent automated AI R&D or the achievement of AGI (in 2026-2027), it could be in the sense of routine AI R&D, which then starts the flywheel of accumulating novel deep skills in consecutive versions of AIs, a process that builds towards AGI-level capabilities, including frontier AI R&D, without requiring any new unhobblings or breakthroughs invented by humans as prerequisites. This milestone of automated routine AI R&D seems disturbingly closer than an AI that starts doing frontier AI R&D directly, but it plausibly quickly leads to the same place.
What links here?
- Vladimir_Nesov's comment on Sparks of RSI? by Nathan Helm-Burger (14 Mar 2026 18:35 UTC; 16 points)

Vladimir_Nesov 10 Mar 2026 1:02 UTC
4 points
0
in reply to: silentbob’s comment on: silentbob’s Shortform
None of this helps with automatically acquiring deep skills like playing good chess or fluency in a novel topic of math, and so these aren’t the stright lines on graphs directly relevant to crossing the AGI threshold, full automation of civilization.

Humans don’t know how to automate learning of arbitrary deep skills in an AI that only come up post-deployment, but can manually add them with RLVR at training time, by developing RL environments, graders, and tasks. AI might automate this process not by doing what humans couldn’t and inventing algorithmic advancements for low level acquisition of deep skills, but instead by merely being smart and skilled enough to do all the same things that humans are currently doing to make it work, “manually”. So in principle AI might become able to automatically acquire deep skills if it’s capable enough at routine AI R&D, even if it doesn’t have the capability to acquire deep skills at a low level, the way humans do, and doesn’t have the capability to invent substantial algorithmic innovations that humans haven’t invented yet. Some of the straight lines on graphs are relevant to when this might happen, and so indirectly they are relevant to crossing the AGI threshold.

I don’t think in-context learning or even true continual learning with anything like the current methods can automate acquisition of deep skills at a low level, because only RLVR currently works for that purpose, context persistence is essentially unrelated. But these things might get AIs to the level of capability where they can do the same things as the humans who set up the ingredients for task-specific RLVR.