Vladimir_Nesov

Karma: 38,985

Vladimir_Nesov 27 Jun 2026 14:44 UTC
3 points
0
in reply to: Petropolitan’s comment on: Vladimir_Nesov’s Shortform
Section 2 of the post talks about generalization in the classical sense, where an RLVRed model that didn’t train on a situation-class magically does well there, or maybe where a model was magically RLVRed on so many situations it never encounters something truly novel ever again. Rather than the sense in which I expect automated RLVR to solve generalization, which is by observing failures of generalization (that are themselves sufficiently comprehensible with the current model) and producing RLVR training data that addresses them individually, to be included when building the next model. Thus jaggedness is addressed by slowly working on fixing the gaps as they are encountered, when they happen to be near enough to the edge of the existing skills.

(Incidentally, a lot of what currently contributes to the felt impression of jaggedness, rather than this more technical RLVR-and-deep-skills distinction, should also go away as the rising tide of pretraining scale comes in, which after 2026 will happen most prominently by 2028-2029, and to some meaningful further effect by 2031-2032. And then it’s done, no more significant scaling of pretraining for many years.)

Section 1 of the post is some combination of making a distinction between engineering-type skills that can be RLVRed and those that can’t be with current methods, and making a distinction between skills where success is easy to verify and those where it takes too much time, requires experimental feedback, or can’t be packaged in a reproducible enough way to be usable with RLVR. The method has a scope of applicability and doesn’t magically work for everything. This doesn’t mean that the harder engineering-type things can’t be understood as a practice of more modular skills that can be directly RLVRed. You can turn any engineering-type activity into a collection of textbooks with exercises that are individually of manageable scope.

So I take Sections 1 and 2 of the essay as being importantly mistaken (about a priori vs. adaptive generalization; and about modularity of engineering-type skills), while Sections 3 and 4 are more promising, but not predictive of timing.

Vladimir_Nesov 27 Jun 2026 5:35 UTC
3 points
1
on: White House Will Ad Hoc Decide Who Can Individually Access GPT-5.6

I am happy that this suggests our policy is not primarily ‘try to murder or cripple Anthropic in particular,’ or at least that they will not be too hypocritical around that.

Whether GPT-5.6 Sol (Sol 5.6?) or Fable 5 gets to be explicitly cleared for release first (outside a strict whitelist, and not just via Anthropic implementing KYC) will be the next piece of evidence about this. The export control directive currently applies to Fable 5, but not to Sol 5.6, and this distinction might persist. Even as it seems like a silly possibility, that’s not obviously a good heuristic for ruling it out. (It’s possible OpenAI adopted the Sol/Terra/Luna naming convention in part to make it more likely that any Mythos-like restrictions get appropriately scoped to Sol 5 models only, rather than to GPT-5.6+ as a whole.)

Vladimir_Nesov 26 Jun 2026 22:28 UTC
5 points
0
in reply to: Petropolitan’s comment on: Vladimir_Nesov’s Shortform
Dwarkesh Patel is unfortunately not insisting on the distinction between novel deep skills, and the stuff that in-context learning can handle (either simple skills or existing deep skill that are already in the weights). Because of this distinction, sample efficiency for novel deep skills can’t be usefully found in in-context learning, or in continual learning methods that repackage the capabilities of in-context learning using some sort of recurrent state that enables functionally unlimited contexts (memory/skill note directories explicitly maintained by the model also fit this description).

Some hypothetical continual learning methods of the future might be good enough for learning deep skills, better than the present day in-context learning is, but this is not a solved problem, it’s not something that can be expected to start working within some predictable amount of time. Dwarkesh Patel is caught up in essentially speculating how that could happen. One thing he mentions is on-policy self-distillation, which is an obvious enough idea to try, at least after you know about OPD itself, so every month it’s not demonstrated to be RLVR-level capable (when used outside RLVR) is evidence it’s not ready to soon become important for continual learning of novel deep skills (rather than for something else, like augmenting RLVR with process level feedback, maybe as a step in some recipe for RLVR training of multiple teacher models intended for eventual OPD into the target model).

Another thing he points to is “dreaming” that seems to be about model based RL (the example he gives is EfficientZero), but that has been a very long-running research direction, so the fact that it doesn’t work with LLMs yet is already significant evidence that it can’t be expected to start working within a short predictable amount of time. If you ignore all the technical distinctions, model based RL is somewhat similar to learning in self-constructed RL environments in RLVR, but focusing on RLVR specifically (rather than on some other RL shaped thing with constructed environments) is crucial because RLVR is the only thing that already works for teaching deep skills to LLMs. And RLVR is known to work well specifically when it happens as part of the overall model training/development/building process, not necessarily in long standalone sequences of post-deployment fine-tuning.

With current methods, sample efficiency in acquisition of deep skills can only be achieved by constructing the training data for RLVR (tasks/environments/graders), based on understanding the difficulties posed by the absence of specific novel deep skills (and the understanding of these difficulties can be obtained sample efficiently with in-context learning). This is impractical for human AI engineers to do manually for every little on-the-job thing, but can become practical once models can do that automatically on their own (which the remaining scaling runway likely allows).

Vladimir_Nesov 24 Jun 2026 20:07 UTC
10 points
0
on: Fable in Shackles
Non-US Non-Chinese AI companies need a currently-implausible level of funding to keep scaling at the frontier level, while the Chinese AI companies need large scale-up systems that aren’t being sold to them. It wouldn’t matter if they have enough FLOP/s in total in the form of servers that can’t fit large models, H100/H200/B200 (NVL8) rather than GB200/GB300 Oberon (NVL72) or future Kyber racks. It remains unclear when Huawei will actually start producing their large scale-up systems (CloudMatrix/Atlas) in important quantities, let alone in quantities that would match the frontier US compute, while maintaining a similar level of system-level performance (for individual scale-up systems needed to run large models).

So far, the Chinese models remain at a roughly 1T total params scale, which fits in the older 8-chip servers. But the 2025-2026 Oberon racks enable 20-25T total param models, then 2028 Kyber racks enable 400T total param models, and 8x Kyber Feynman systems of 2029-2030 enable 1-3 quadrillion param models. These models absolutely can’t run on the 8-chip servers currently used to run the 1T total param models, no matter how many such servers you have. My guess is that it’s actually useful to scale models almost as far as that hardware allows, and it’s not as expensive as it sounds once the large scale-up systems are available.

It is also unclear whether Chinese labs, or American open-source labs like Reflection and Arcee, can come up with algorithmic innovations to leapfrog the closed-source frontier.

You can’t sustainably leapfrog the closed-source frontier with algorithmic innovations, because the closed-source frontier will leapfrog you back with the same algorithmic innovations, only it’ll scale those innovations with more compute and to larger models, resulting in even higher quality.

Vladimir_Nesov 24 Jun 2026 17:01 UTC
7 points
0
in reply to: Petropolitan’s comment on: Model Size Scaling in 2023-2031
The Farseer paper does a lot of theoretical extrapolation from experiments with up to 3e21 FLOPs (and it really doesn’t like isoFLOPs, though the raw experimental data was published, so it’s possible to extract them). I expect compute optimal D/N ratio to be somewhat stable, merely on priors, and that is the sense in which I’m following Chinchilla scaling (the 240 tokens/param at 30x sparsity is quantitatively very different from Chinchilla’s 20 tokens/param). So I’m not engaging with the Farseer vs. Chinchilla debate that the Farseer paper is having, about the functional form of how loss or BPC depend on D and N, because I reject that framing and don’t reason from the considerations relevant there, even as my conclusions side with Chinchilla. Instead, I’m looking at whatever experimental data about the D/N ratio I can find, and running with it all the way to 2e29 FLOPs (modifying it with the split-the-shortfall rule once the data runs out, which I think is plausible with up to 4-5 epochs of repetition).

Arbitrarily high sparsity seems useful with unlimited data, see Figure 11 from this paper (the horizontal axis is log-sparsity; it looks like infinite sparsity gives an infinite compute multiplier). But sparsity drives the D/N ratio higher, seemingly in step with the compute multiplier. That is, 8x sparsity gives a 3x compute multiplier over dense, but also drives the compute optimal D/N ratio 3x higher compared to dense; and 30x sparsity gives a 6x compute multiplier, driving the D/N ratio 6x higher over dense (see Figure 12, left for the relevant isoFLOPs). Thus very high sparsity would ask for even more data, and so by the split-the-shortfall rule would need even more epochs of repetition (the 2031 model from this post already reaches 4 epochs). When data is scarce, high sparsity gets less helpful.

For the Chinese models, which target the 1T total params generation (pre-Oberon) scale-up systems for inference, it makes sense to fill all the available space above the active params up to about 1T total params with sparsity. But if they had more active params with the same number of total params, the quality would’ve been higher (as would the cost of pretraining and inference). So while keeping the total params count fixed, lower (rather than higher) sparsity gives better quality (and needs less data for a given amount of pretraining compute). It’s only when you keep the active params count (or pretraining compute) fixed, and the total params are not constrained, that higher sparity gives better quality. The reason to go for high sparsity (while observing the roughly 1T total params upper bound) is insufficient pretraining compute, or inability to bear the inference cost of models with more active params, and both these factors are plausibly why the Chinese models have so few active params and therefore high sparsity.

Vladimir_Nesov 23 Jun 2026 3:41 UTC
12 points
0
in reply to: StanislavKrym’s comment on: Model Size Scaling in 2023-2031
Input token cost scales with active params, essentially doesn’t depend on total params (the same as the FLOPs of pretraining), and goes down over the years as FLOPs get cheaper. Output token cost is determined by KV cache per token times average context length. KV cache per token scales with model dimension, which scales with square root of active params or even slower (if more active params translates to more layers or more experts per layer), that is it scales significantly slower than the active params themselves do. Of course, output token cost can’t actually get below that of the input tokens because the output tokens are HBM bandwidth bound (for big models), so their cost (not necessarily price!) remains at least 2-3x higher, but with longer contexts they become even more expensive compared to the input tokens. So at a sufficient context length, the scaling trends play out if we keep the context length fixed and vary the active params of models, with the output token cost increasing slower than the input token cost as the number of active params increases.

Most tokens are input tokens, so their pricing matters more for the AI company. In principle, output tokens could be priced with zero gross margin, they just shouldn’t be generated at a loss. The cost of the output tokens is kept at a healthy 5-6x compared to input tokens by increasing the allowed context length.

So let’s say the 2026 model with 1.3T active params is literally Mythos 5 and it’s priced at $10 per 1M input tokens. That’s already with a gross margin that might be above 70%, since the 70% figure could probably anchor Opus 4.6 price, which is $5 per 1M input tokens, exactly 2x lower than Mythos 5 price, and Mythos 5 probably doesn’t have more than 2x the active params than Opus 4.6, since it probably didn’t use more than 4x the compute of Opus 4 in pretraining, and both plausibly use compute optimally sized numbers of active params (maybe Mythos 5 has more sparsity than Opus 4) without yet running out of unique training data (active params scale with the square root of pretraining compute). Which is to say, the $10 per 1M input tokens price of Mythos 5 probably has significant room for coming down even today.

The 240T total params model of 2028 has only 7.9T active params, 6x more than the 1.3T active params of the 2026 model, and it runs on Rubin instead of Blackwell. GB200 produces 5e15 FP4 FLOP/s per compute die, Rubin produces 17.5e15 FP4 FLOP/s per compute die, GB200 rack needs 140 kW, Rubin Oberon rack needs 225 kW. The cost per year of a 1 GW datacenter (not just the racks) might be going up 20%, so the cost of the datacenter time per compute die goes up maybe 90%, but the FP4 FLOP/s go up 3.5x, and so the FP4 FLOPs in Rubin are 1.8x cheaper than the FP4 FLOPs in GB200 (the 2028 model needs Kyber rather than Oberon, but let’s assume the cost of time per compute die isn’t too different). Thus the 6x difference in active params becomes a 3.3x difference in cost. The price for the 240T total params model of 2028 might be $33/$165 per 1M input/output tokens. (The output token price is adjusted as appropriate by increasing the allowed context length, if it would otherwise try to get below 5x the input token price. And this is plausibly still at a more than 70% gross margin, so could be cheaper actually.)
What links here?
- Vladimir_Nesov's comment on Fable in Shackles by ykevinzhang (24 Jun 2026 20:07 UTC; 10 points)

Vladimir_Nesov 23 Jun 2026 2:16 UTC
8 points
0
in reply to: bhalstead’s comment on: Model Size Scaling in 2023-2031
First-party compute is distinct from serving inference via clouds (IaaS as opposed to TaaS compute). The quote you gave (at 11:30 in the podcast) discusses IaaS plus TaaS, while I talk about just IaaS (which is more relevant for R&D and training). In the same interview, Dylan Patel also says Anthropic will get to 2+ GW end of 2026 (at 1:12:10 in the podcast, the link I gave in the post for the 10 GW claims), which likely refers to IaaS only.

I’m basing the 3-4 GW figure on multiple less-specific clues and interpolation. For Anthropic, the general IT power of compute trend of about 3x per year, the claim of 2+ GW end of 2026 probably implies 1+ GW end of 2025, then there’s 10 GW end of 2027, 1 GW of TPUs in the 2026 buildout, more Trainium in 2026, so I think 2+ GW end of 2026 likely resolves to 3 GW. For OpenAI, there are contracts with Oracle for Blackwell (see the table in this section of the TPU post), which is 2025-2026 incremental compute, and compute at the end of 2025 is 2 GW. Plausibly OpenAI gets more than 4 GW end of 2026, but then it might rely on first-party compute to serve tokens more than Anthropic, and so use less of it for R&D and pretraining. Also, if OpenAI gets back to the same 10 GW at the end of 2027 as Anthropic, the possible difference over 2026 balances out, so it shouldn’t be strategically relevant for them when sizing pretraining runs.

Vladimir_Nesov 23 Jun 2026 1:40 UTC
4 points
0
in reply to: jsd’s comment on: Model Size Scaling in 2023-2031
A more precise value is 27.6T. It’s an upper bound, so I rounded down. It’s a difference of 2%, at this level I might use rounding inconsistently, and depending on context I sometimes keep just 1-2 significant digits.

Vladimir_Nesov 23 Jun 2026 1:28 UTC
13 points
0
in reply to: Lanthimum’s comment on: Model Size Scaling in 2023-2031
The compute scaling is much less significant than a 24x difference in model size would imply under the all-else-equal scaling assumptions, which is 24x squared. It’s instead going from 1.3e27 FLOPs (probably Mythos 5, possibly GPT-5.5 as well, or its base model will get there with more training in GPT-5.6) to 2e28 FLOPs, a 15x increase. Then there’s the change from 8x sparsity to 30x sparsity, which should give a 2x effective compute boost, to the total of 30x (3x boost from 8x sparsity compared to dense, 6x boost from 30x sparsity, see Figure 11 from this paper). But also the 2026 model has enough data, while the 2028 model is under a 4.5x data shortfall, so unclear how much of the effective compute boost survives. Maybe the 2028 model has 20x more effective compute in pretraining than the 2026 model.

On the other side of the 1.3e27 FLOPs, a 20x decrease gives 6e25 FLOPs, 3x less than late 2023 to very early 2024 compute that probably trained GPT-4o (though the model was likely significantly smaller than compute optimal). This same pretrain probably became o3 and then GPT-5.1 later in life. Perhaps the GPT-4o base model being smaller/overtrained cancels out the difference in compute, so it might be at the level of a compute optimal 8x sparsity model pretrained with 6e25 FLOPs. And GPT-5.1 was maybe around the level of Sonnet 4.5 if we try to ignore the differences in how they were post-trained, so the 2028 model might be stronger than the 2026 model by about as much as Mythos 5 is stronger than Sonnet 4.5 or GPT-5.1.

Then the 2031 model is using merely 11x more pretraining FLOPs than the 2028 model, and uses the same 30x sparsity (so there is no boost to effective compute from more sparsity), and has a 15x training data shortfall instead of 4x. So the difference between the 2028 and 2031 models is maybe 1.5x smaller than the difference between the 2026 and 2028 models (comparing the effective compute on the logarithmic scale), unless much larger pretrains have some currently-unknown advantages not captured in their token prediction loss. Perhaps the 2031 model is smarter than the 2028 model by about as much as Mythos 5 is smarter than the average of Sonnet 4.5 and Opus 4.6.

(Standard caveats, algorithmic advances can happen suddently at any time, any predictions about the current paradigm are only relevant as long as the current paradigm itself remains relevant.)

Model Size Scaling in 2023-2031

Vladimir_Nesov22 Jun 2026 23:07 UTC

72 points

15 comments12 min readLW link

Vladimir_Nesov 21 Jun 2026 22:23 UTC
11 points
4
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform

identify the considerations that matter most, and ignore the rest … ask yourself the right questions

Working through details helps with being able to notice and use further clues, improving the ability to ground the models in reality. This works regardless of whether these are the right considerations that matter most, and even regardless of whether the initial detailed models are any good. It’s just impractical to notice and think about new details if you haven’t thought about related things before, especially for something observed tangentially rather than in the spotlight. Existing fluency buys a lot of sample efficiency.

Vladimir_Nesov 21 Jun 2026 22:01 UTC
11 points
−7
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform

Meanwhile in AI safety, our best plan is still to defer to humans on questions like “What should ASI do with our lightcone?”

I think for now it’s reasonable to have ASIs mostly work on converting the lightcone into potential computronium (if we get any say in what ASIs do), planning for efficient production of compute across astronomical time, if humans get to remain alive indefinitely without going increasingly insane, with some checkpoints and instantiation infrastructure to be able to do more. We might eventually figure out how to grow up sufficiently to be able to do something reasonable with a share of this compute, so the optionality of waiting and preparing for such decisions (constrained by some guardrails) seems reasonable, even when ultimately these are human decisions. If some compute must be expended sooner (and otherwise lost forever) for physics/engineering reasons, it might be good for such guardrails to allow less considered, or less individual use of it. There’s going to be enough of such compute to fuel even very ambitious (if poorly considered) takes on long reflection.

If instead ASI gets to use most of our lightcone without human input, and without the ASI itself being constructed in a considered way based on human input, that is permanent disempowerment (even if somehow there is no extinction). ASI gets to do something with its lightcone, it’s not our lightcone anymore. Unclear if the ASIs that should do something with our lightcone should be constructed entities clearly separate from us, or the result of uplifting ourselves individually, after it becomes a lot clearer how that could possibly be a meaningful proposition and also not a disaster. But that requires a lot of time to figure it out in a legitimate way, so that it’s not exogenous ASIs dictating how to do things, before we are ready to shape such decisions.

Vladimir_Nesov 20 Jun 2026 14:47 UTC
32 points
0
on: Vladimir_Nesov’s Shortform
SemiAnalysis claims HBM for Rubin Ultra was downgraded from 16-Hi to 12-Hi stacks. The 16-Hi HBM was going to be HBM4E, with 4 GB per DRAM die, so 48 GB for 12-Hi stacks. In Rubin Ultra, there are 4 stacks of HBM per compute die, and in Kyber racks there are 144 packages with 4 compute dies each. Thus 192 GB instead of 256 GB per compute die, 768 GB instead of 1024 GB per package, and 110 TB instead of 147 TB per Kyber Rubin Ultra rack. This has implications for practical model sizes of 2028-2029 (for the largest models), they just got 25% smaller, all else equal. In the rest of the post, I discuss a likely change in Feynman that halves HBM per rack compared to what I recently estimated, giving merely 737 TB per 8x Kyber scale-up system in the second year of Feynman. This also makes it more likely that the 8x Kyber scale-up systems (589 TB HBM) are available in the first year of Feynman, rather than only in the second year.

First year Feynman (on the roadmap for 2028, most of the buildout completes over 2029) will probably get back to 16-Hi HBM4E stacks. But Feynman might use a different arrangement of compute dies (original slide deck), so that there are only 2 HBM stacks per compute die instead of 4. It’s unclear if this change happens with the first year Feynman, since it’s listed as a 2029 design for HBM5 in the KAIST (Joungho Kim) HBM roadmap (from Jun 2025), while the first year of Feynman is 2028. So either the first year Feynman already has 2 HBM stacks per compute die, but it’s HBM4E, or the first year Feynman still has 4 HBM stacks per compute die (as in Rubin Ultra). Another question is whether the version of Feynman with 2 HBM stacks per compute die still has 576 compute dies per Kyber rack (in 144 packages), or moves to 1152 compute dies per Kyber rack (in 288 packages). The latter might ask for infeasible power density per rack, but the former reduces the number of HBM stacks per Kyber rack from 2304 (for Rubin Ultra) to 1152 (for Feynman that adopts 2 HBM stacks per compute die).

Thus first year Feynman (2028) might have 147 TB per Kyber rack if it’s 4 16-Hi HBM4E stacks per compute die or 73 TB if it’s 2 stacks per compute die (576 compute dies in 144 packages per rack, in both cases). And second year Feynman (2029) that uses HBM5 (5 GB per DRAM die) might have 92 TB per Kyber rack if it’s 2 16-Hi stacks per compute die and 144 packages per rack (4 compute dies and 8 HBM stacks per package), or 184 TB per rack if it’s 288 packages.

Since reducing HBM capacity relative to the previous generation doesn’t look good in a product, the largest practical scale-up offering should have at least 2x the number of Kyber racks compared to the previous generation when the move to 2 HBM stacks per compute die happens, if it doesn’t simultaneously double the number of compute dies per rack. So either there’s a 16x Kyber system for second year Feynman (this wasn’t announced), there are 2x more compute dies per Kyber in second year Feynman (making the power density even more alarming), or the first year Feynman will already offer the 8x Kyber system and have 2 HBM stacks per compute die.

My guess is the last possibility, meaning 73 TB per Kyber rack (rather than 147 TB, and down from the 110 TB of Kyber Rubin Ultra), but with 589 TB per 8x Kyber system (thus overall it’s still up from 110 TB of Kyber Rubin Ultra) for first year Feynman. And then the second year of Feynman (2029, buildout completes mostly over 2030) merely upgrades from HBM4E to HBM5, keeping 8x Kyber as the largest practical scale-up offering, meaning 92 TB per Kyber rack and 737 TB per 8x Kyber system.

(This is 2x lower than the 1.18 PB and 1.47 PB per system I previously expected, but 10 GW of pretraining compute still predicts 1 quadrillion total params, and that remains borderline practical with these systems. Though it probably won’t tolerate FP8 for inference, might really want the contemporary TPUs for RLVR, and sparsity for the largest models of 2030-2031 might need to remain below 30x.)
What links here?
- Model Size Scaling in 2023-2031 by Vladimir_Nesov (22 Jun 2026 23:07 UTC; 72 points)

Vladimir_Nesov 19 Jun 2026 20:49 UTC
73 points
23
on: Vladimir_Nesov’s Shortform
AGI-pilled people often don’t take seriously either the possibility of many years of RSI-without-ASI, or the possibility of a sudden ASI at some point. I think the baseline scenario involves both, and ignoring either one of them predicts very different effects of possible policies.

The basic case of recursive self-improvement is self-building, an LLM autonomously building a new version of an LLM like itself, without necessarily making important non-routine improvements, at which point the extent of self-improvement becomes quantitative. This is a project involving verifiable tasks, that LLMs seem to be very on track to become capable of being trained for using RLVR (in a broad sense, with graders that can invoke LLMs). The LLMs of 2026 are probably about 1T active params, 10T total, trained for 1e27 FLOPs with 300 MW of compute capacity. The LLMs of 2031 might be about 40T active params, 1,000T total, trained for 2e29 FLOPs with 10 GW of compute capacity, 10x more expensive per token, and slightly faster than the LLMs of today. The next big model scale-up comes with Nvidia Kyber racks in 2028 (when most of the buildout happens; a bit of it starts in 2027), so it’s likely self-building starts working then. And then the 8x Kyber Feynman systems of 2029 or 2030 might enable models capable of self-improvement to a more meaningful degree than mere self-building.

Being AGI-pilled about LLMs paints the picture of not just AGI as a normal technology, but RSI as a normal technology. Prosaic RSI of preparing new training data, including new RLVR tasks and environments, and then building the next LLM solves generalization by introducing the out-of-distribution situations encountered in practice into the training data for the next model. This is quite slow when using the current cookbook, even if the process is fully automated, performed by the LLMs themselves. Yet this promises full automation (and automation-of-automation) for all engineering-type or otherwise verifiable tasks, no matter how specialized. Thus we get AGI, which is slow at adapting to new situations, but capable of general learning and fast at executing on what it already knows. This doesn’t automatically lead to ASI, and the RSI process isn’t inherently destabilizing.

Not being ASI-pilled is then a giant blind spot, where you expect a “smooth exponential” that is on-trend for the LLMs, a controlled chain reaction of prosaic RSI, possibly leading to some sort of similarly smooth industrial explosion, but not to a software-only intelligence explosion, where cognition goes supercritical. I think the valid point is that prosaic RSI with LLMs doesn’t predict ASI any time soon, and thus RSI can continue smoothly for many years.

But while this is happening, nothing prevents a breakthrough (at any time) that makes learning of deep skills fast rather than slow, at the speed comparable to token generation rather than model releases, cutting any plans built on the foundation of the “smooth exponential” short. Hundreds of gigawatts of global AI compute in the 2030s can rapidly scale any such breakthrough to overwhelming effect, with all the engineering details taken care of by the LLMs, after some steps of prosaic RSI if that’s necessary for them to really understand what needs to be done to scale the breakthrough. Not expecting this possibility risks failing to prepare for it (and then suddenly it’s too late). In contrast, not seeing the possibility of prosaic RSI merely delays when the policies about the ASI danger (apart from shutting it all down) start talking about the relevant context in which they need to be able to work.
What links here?
- Vladimir_Nesov's comment on Model Size Scaling in 2023-2031 by Vladimir_Nesov (23 Jun 2026 1:28 UTC; 13 points)

Vladimir_Nesov 18 Jun 2026 23:18 UTC
3 points
0
in reply to: Paragox’s comment on: Vladimir_Nesov’s Shortform
Pretraining compute (and unique data still being plausibly available in sufficient amounts) constrains what the desirable active param counts are. And inference isn’t that expensive when there’s enough demand, the actual prices don’t require small models even at fat margins (which is also why future quadrillion param models remain affordable). This is the reason I expect 1T active param models from 300 MW of pretraining compute, and these models could be served as the largest offerings directly (the smaller models could be obtained by pretraining distillation).

Since very high sparsity on top of 1T active params is impractical in RLVR and decode (in 2025-2026), in principle there could be high sparsity pretrains that then get distilled into lower sparsity pretrains with about the same number of active params (and the high sparsity pretrains then never get RLVRed or released). But that takes 2.3x more compute in pretraining (or asks the models to be smaller, which might be worse than just training the lower-sparsity model directly), 2x more wall-clock time, and both pretraining runs at the new level of scale have to succeed, one of them at high sparsity. So I don’t particularly expect this to happen for the largest models while the largest models keep getting much larger, and I don’t expect this to happen for the smaller models, since there are larger models to pretraining distill them from. This kind of thing in principle could’ve happenned sometime after 2032 when the models no longer keep getting larger, and the largest models remain at a scale where the previous training runs were already perfected. But after 2032 the scale-up systems get so large that even high sparsity models fit in a scale-up system, so there is still no reason to do the distill-to-lower-sparsity dance. Thus I expect this to never happen at all.

OpenAI’s apparent lack of 5.5 scale vs Mythos/Fable, which still seems to me most obviously occam’s razored away by Anthropic having TPUs at scale and OpenAI not

I don’t know if GPT-5.5 is meaningfully smaller than Mythos (especially in active params). It might just be undertrained for now. The Mythos situation additionally illustrates the wisdom of frogboiling. TPUs would only be relevant compared to Trainium 2 Ultra if it’s TPUv7, and there likely wasn’t enough of it at Anthropic when Mythos Preview was RLVRed. For pretraining, it wouldn’t much matter if it was H100s.

Vladimir_Nesov 18 Jun 2026 20:30 UTC
6 points
0
on: AI #173: AI Pauses

also the words are so slopirific and full of corporate buzzwords that my brain kept bouncing off of them … dealing with AI slop has made my brain unwilling to read other forms of slop as well, and I endorse this as healthy

This is similar to reading in a foreign language that you don’t know very well (even when you recognize all of the words), and could be a signal that you need to focus more on texts like this to gain fluency. The feeling of incomplete fluency in a dialect is different from what you get when reading something that’s poorly thought-out, even as both can be the case.

Gaining specialized fluency is often useless (the same as with languages), but enables you to discern enough aspects of meaning and domain specific references that some pieces of apparent slop starts reading as making meaningful claims instead (if perhaps expressed in a needlessly sloppy and unhelpful way). Or to be able to skim more efficiently, quickly figuring out that there’s a nugget of wisdom on page 17. Other things are inherently slop, but even then can be useful for improving the comprehension of the domain specific dialect.

Vladimir_Nesov 18 Jun 2026 19:01 UTC
17 points
0
on: Contra Pace on When to Apologize
Apologies acknowledge a harm. Understanding is a key input to belief, which is a key input to decision making, which gets more relevant when that same person’s decisions played a role in the harm. But like Bayesian probabilities, harm makes sense acausally, and the influence can originate from norms and values, not just concrete humans. So being sorry someone died from disease acknowledges a harm, and can still be relevant to the extent it channels cognition of the norms and values that would try to prevent that harm if given a chance. The concrete human channeling this idealized judgement of values doesn’t need to change policy for the apology to be meaningful and predict what the influence of the values would be under the right circumstances, when channeled by someone in a position to do something relevant.

When a human doesn’t intend to change behavior, or to compensate people for negative externalities, there might still be norms within that human that judge this to be a harm (with the human being influenced but not fully controlled by the norm), including coalition-reifying norms that could act through other humans or institutions, and understanding the possible behaviors at those other sites that channel the influence of the norm benefits from knowing whether the norm makes the judgement of harm having been done. So even an apology that doesn’t come with an intention of compensation can still be meaningful and truthful in this sense, as a claim about decision-making cognition of an actor that is not the human delivering the apology. Similarly, future opportunities to compensate or to enact a relevantly changed policy are not always available, but the apology can still express a judgement about that single past instance, the way a Bayesian probability of a one-time event can still be quantitatively wrong.

(A claim of acknowledging a harm or of being influenced by a norm that aspires to make that judgement can of course be false, so observations of changed behavior or compensation are very useful evidence. But this is about practical ways of judging the truth of an apology, or extracting some use out of it, not something inherent to the meaning of an apology.)

Vladimir_Nesov 14 Jun 2026 0:56 UTC
4 points
2
in reply to: MichaelDickens’s comment on: MichaelDickens’s Shortform

too easy to get them to say what you want to hear

I really don’t. It’s a meaningful statement, but not a truthful framing.

Vladimir_Nesov 13 Jun 2026 23:06 UTC
2 points
0
in reply to: the gears to ascension’s comment on: the gears to ascenscion’s Shortform
It’s an important part of what’s going on, but not “the reason it (the glass) is able to learn to talk”.

Vladimir_Nesov 13 Jun 2026 22:39 UTC
2 points
0
in reply to: the gears to ascension’s comment on: the gears to ascenscion’s Shortform

anywhere the glass (the underlying model) was ingesting text about a real human, the glass contains an image of that human

A coherent text about cattle, or abelian groups, is not about some human, even if the text was written by a human.

the reason it (the glass) is able to learn to talk is the same reason it (the glass) contains the dynamical image of an authorial person

Construction of a chatbot persona with a consistent personality must reference the writer, but it’s not a good explanation for how models can understand things and put that understanding in writing, when the understanding is about some non-human situation and not particularly about the writer. Observing text that coherently describes consistent situations is a common reason between how LLMs understand what they are talking about, and how humans understand what they are talking about. A common explanation for both, not one unexplained thing explaining the other via imitation.

Vladimir_Nesov

Model Size Scal­ing in 2023-2031

Model Size Scaling in 2023-2031