Vladimir_Nesov

Karma: 39,042

Vladimir_Nesov 7 Jul 2026 4:22 UTC
3 points
0
in reply to: dr_s’s comment on: The case for the fleshman

This is technically “engaging with the outgroup”—the craziest member is still a real member!

This is not centrally the case. You don’t need to go for the median to draw a sensible boundary and then retreat from it to the inside, with some margin. There will remain a lot of variation, on both content and quality, but you won’t risk the failure modes of non-centrality.

the most sane, thoughtful representative of the idea you’re arguing against, someone so well articulated

The criteria that grade arguments as sane/thoughtful/well-articulated should be suspect, because something might be such from your point of view, but not from inside the target worldview. If you optimize too strongly, you risk goodharting on your alien proxy measures of quality and pick a non-central representative as a “steelman” source.

so well articulated that you have a hard time finding fault in their version of the idea at all

Failing to find fault is very different from understanding an idea, digesting it well enough to keep developing it on your own. The purpose of steelmanning, in the (more useful, I think) distortionary variant of its meaning, is to build your own ideas regardless of whether they accurately represent and should be attributed to their original source/inspiration. This doesn’t work at all if you merely “have a hard time finding fault”.

Vladimir_Nesov 7 Jul 2026 1:02 UTC
6 points
1
on: Be less open-minded: Against uncalibrated open-mindedness
This is a false dichotomy that can be resolved by coloring understanding with epistemic status. It’s prudent to grow understanding of confused and contentious things, of points of view that seem alien, as long as you don’t forget they are not your own (at least initially), and don’t let them take over all of your time. Understanding things well requires taking them seriously, but carries no obligation to believe or endorse what you come to understand.

This understanding then lets you speak its language, to find signal in other things that otherwise would seem like noise or senseless conflict. Occasionally, it turns out to contain valuable lessons, something worth endorsing. Crucially, this is the only practical way to defeat path dependence, to make your own epistemic luck.

Vladimir_Nesov 6 Jul 2026 2:41 UTC
4 points
0
in reply to: StanislavKrym’s comment on: Reevaluating AI-2027: timelines, takeoff, alignment and China
Active and total param counts are very different things, they can’t be used interchangeably. Smaller models with about 30x sparsity are already common, and the largest models will likely start using about 30x sparsity in 2028+.

dense-equivalent parameters

(I apologize for the previous version of this comment where I got confused and estimated this incorrectly. This is the fixed version.)

My scaling assumptions (mostly based on this paper) say that the effective compute boost from sparsity is about the same as the increase in compute optimal D/N ratio over dense models, both are about 3x for 8x sparsity, and 6x for 30x sparsity. That is, a compute optimal model with 8x sparsity would have the quality of a dense model that uses 3x more compute at a 3x lower D/N ratio. Using 3x more compute multiplies active params by square root of 3, and using a 3x lower D/N ratio additionally multiplies active params by square root of 3, so the number of active params gets 3x higher in total. The dense-equivalent params for a 8x sparse model are 3x more than its active params (and 2.7 less than its total params). And the dense-equivalent params for a 30x sparse model are 6x more than its active params (and 5x less than its total params).

For the 1.3T active params 8x sparse 2026 model, the dense-equivalent number of params is 3.9T. That is, a compute optimal dense model with 3.9T params will achieve about the same token prediction quality as the 8x sparse model with 1.3T active params (and 10T total), but the dense model would need 3x more raw compute (both its total pretraining and 1M of its input tokens would be 3x more expensive).

GPT-N-pro models

My impression is that there are no distinct GPT-N Pro models, these are just ways of running GPT-N models with parallel compute to improve quality (aggregating from multiple inference traces).

Vladimir_Nesov 6 Jul 2026 1:12 UTC
8 points
0
in reply to: Coagulopath’s comment on: Reevaluating AI-2027: timelines, takeoff, alignment and China

does anyone know how reliable this 2T/10T claim is?

(I don’t know where StanislavKrym got the 2T Opus claim.) My expectation was 3T total params for Opus 4 based on Trainium 2 Ultra systems having about 6 TB of HBM and no support for FP4, so that the weights could fit in half a scale-up system. But then there’s a Musk tweet that claims “Opus” is 5T params.

My estimates from pretraining compute would predict 7T total params for Opus 4 (if it’s trained with about 100K H100s) and 10T for Mythos 5 at 8x sparsity (if it trained with about 200K H100s). But 10T for Mythos 5 in 2026 fits in Oberon racks, while 7T for Opus 4 would have more trouble in 2025, when the buildout of Oberon racks was insufficient, and Anthropic had to make use of their Trainium 2 Ultra datacenters. HBM in Trainium 2 has 2.9 TB/s BW and 96 GB capacity, 33 ms to fully read, a bit worse than even H200, so in the framing of my post only inference deployments in 1 scale-up system would be acceptable (using more would make token generation too slow). Thus I think the 10T asked-for by pretraining compute is fine for the big 2026 model (such as Mythos 5), but 7T is a bit too much for the big 2025 model.

Also, a Dec 2024 Anthropic post claims that the Rainier datacenters would provide “five times the computing power (in exaflops) used to train our current generation of leading AI models”. The reference to the “current generation of leading AI models” is ambiguous, since only Opus 3 was released back then, but Opus 4 was probably already pretrained, so the claim might be referring to it. At that stage in the planned Trainium 2 buildout for Anthropic, the Rainier compute might’ve referred to 400K chips, which in FP8 would correspond to about 250K H100s (1.3e15 FP8 FLOP/s per Trainium 2 chip compared to 2e15 FP8 FLOP/s per H100 chip). A fifth of that is 50K H100s, 2x less than my assumptions for the big model of 2025 and 4x less than my assumptions for the big 2026 model. Since model size scales with square root of pretraining compute, Opus 4 might be exactly 2x smaller than Mythos 5 just from pretraining compute considerations (if it has the same sparsity). This fits the Musk claim about 5T total params for Opus (when assuming 10T total params for Mythos 5), and since the cost of input tokens is proportional to the active params count, this also lines up with the $5/$10 price for Opus 4/Mythos 5 input tokens. Though 5T still seems too much for inference on Trainium 2 Ultra, unless the weights are stored in FP4 in HBM during deployment (which is likely possible). It’s either that, or the sparsity is lower than 8x for Opus 4, or my D/N ratios in scaling laws are off (which is very possible).

I’m not saying I don’t believe you, I’m just curious.

(Starting to believe something when you don’t know how reliable it is seems like an obviously bad default stance.)

Vladimir_Nesov 5 Jul 2026 16:54 UTC
6 points
2
in reply to: Richard_Kennaway’s comment on: The case for the fleshman
Some ideas are meaningful on their own merits, and can be developed according to their own nature. Why engage with the kinds of ideas that are not like that and require an exogenous “primary source” that can’t be grasped? Studying that “primary source” seems like a valid reason, but then it’s not about the ideas.

Vladimir_Nesov 5 Jul 2026 16:40 UTC
3 points
0
in reply to: Richard_Kennaway’s comment on: The case for the fleshman
At some point (after grasping the referent/intent/desiderata), the proponents should become irrelevant to the further study of the ideas, unless you are primarily studying the proponents rather than the ideas.

Vladimir_Nesov 5 Jul 2026 10:48 UTC
5 points
0
in reply to: dr_s’s comment on: The case for the fleshman

what I, specifically, would be like, if I bought into the core premise

Interesting how ITT could be thought of as exploring various modes of being for a person, rather than about the content of various worldviews. Keeping identity small makes this quite an alien perspective (where even a lot of factual understanding fails to graduate to the level of endorsed factual belief). But ITT as modes of being is closer to the framing needed to develop an accurate theory of mind about people who don’t try to keep their identity small.

Vladimir_Nesov 5 Jul 2026 4:36 UTC
8 points
2
on: Reevaluating AI-2027: timelines, takeoff, alignment and China

According to Vladimir Nesov, Mythos’ size could be comparable with Gemini 3 Pro and 3.1 Pro. However, Mythos Preview’s pricing is …

The comment you reference is about total params, not active params. The cost of tokens for big models is determined by active params, not total params (because of the relative scaling of input/output token costs, this should even indirectly apply to the cost of output tokens, despite the technical nature of the cost of output tokens being very different). Additionally, cost is not price, and Dylan Patel says Opus 4.8 is served at a 80% gross margin, so its price of $5 per 1M input tokens might actually rest on the cost of $1 per 1M input tokens.

Nesov estimates that a model from 2028 will have 240T params, causing the horizon to be 8 times longer than the post-o3 trend predicts.

(Just to clarify for other readers, the part about “8 times longer time horizon” is your claim, not mine.) Params (especially total params when a model has 30x sparsity) shouldn’t directly matter for capabilities, my estimate for effective compute was 20x over the big model of 2026 (10T total params, 1.3T active), which might be approximately Mythos 5. I don’t know how you get the “8 times longer time horizon” figure.

Vladimir_Nesov 4 Jul 2026 7:05 UTC
3 points
0
in reply to: Canaletto’s comment on: weightt an’s Shortform

Transparent Newcomb … you can go against the prediction, e.g. one box if it’s empty and two box if they both are full. What Omega offers to those guys, huh?

The situation doesn’t exist for them, as a logical result of their behavior. They also don’t win.

XOR blackmail … There is no way to act on this information and get out of bounds of what is postulated.

The bounds of what is postulated can’t constrain what an algorithm actually does, even when the algorithm gets to know what was postulated in some sense, and is correct in that knowledge in some sense. The algorithm is still the only thing that determines the algorithm’s behavior, it screens off everything else. No other claims or predictions or stipulations get to determine the algorithm’s behavior. It gets to refute even a formal proof about what it does, causing the whole situation where that proof was present or stipulated to never have existed, even if it’s the whole world. The selection effect has no power over the behavior that the abstract algorithm chooses.

Vladimir_Nesov 3 Jul 2026 23:29 UTC
4 points
0
in reply to: Canaletto’s comment on: weightt an’s Shortform
There’s nothing wrong with invalid premises or contrived scenarios. Even if a situation doesn’t exist, you can still ask what happens there, and what should happen there, and what the people who don’t exist should do. Some decisions “cause” agents that make them (situations where they are made) to have never existed (as in Transparent Newcomb), and some decision problems have such decisions as the correct ones (the laws of Nature don’t constrain what an agent gets to do, as an abstract algorithm).

Vladimir_Nesov 27 Jun 2026 14:44 UTC
4 points
0
in reply to: Petropolitan’s comment on: Vladimir_Nesov’s Shortform
Section 2 of the post talks about generalization in the classical sense, where an RLVRed model that didn’t train on a situation-class magically does well there, or maybe where a model was magically RLVRed on so many situations it never encounters something truly novel ever again. Rather than the sense in which I expect automated RLVR to solve generalization, which is by observing failures of generalization (that are themselves sufficiently comprehensible with the current model) and producing RLVR training data that addresses them individually, to be included when building the next model. Thus jaggedness is addressed by slowly working on fixing the gaps as they are encountered, when they happen to be near enough to the edge of the existing skills.

(Incidentally, a lot of what currently contributes to the felt impression of jaggedness, rather than this more technical RLVR-and-deep-skills distinction, should also go away as the rising tide of pretraining scale comes in, which after 2026 will happen most prominently by 2028-2029, and to some meaningful further effect by 2031-2032. And then it’s done, no more significant scaling of pretraining for many years.)

Section 1 of the post is some combination of making a distinction between engineering-type skills that can be RLVRed and those that can’t be with current methods, and making a distinction between skills where success is easy to verify and those where it takes too much time, requires experimental feedback, or can’t be packaged in a reproducible enough way to be usable with RLVR. The method has a scope of applicability and doesn’t magically work for everything. This doesn’t mean that the harder engineering-type things can’t be understood as a practice of more modular skills that can be directly RLVRed. You can turn any engineering-type activity into a collection of textbooks with exercises that are individually of manageable scope.

So I take Sections 1 and 2 of the essay as being importantly mistaken (about a priori vs. adaptive generalization; and about modularity of engineering-type skills), while Sections 3 and 4 are more promising, but not predictive of timing.

Vladimir_Nesov 27 Jun 2026 5:35 UTC
3 points
1
on: White House Will Ad Hoc Decide Who Can Individually Access GPT-5.6

I am happy that this suggests our policy is not primarily ‘try to murder or cripple Anthropic in particular,’ or at least that they will not be too hypocritical around that.

Whether GPT-5.6 Sol (Sol 5.6?) or Fable 5 gets to be explicitly cleared for release first (outside a strict whitelist, and not just via Anthropic implementing KYC) will be the next piece of evidence about this. The export control directive currently applies to Fable 5, but not to Sol 5.6, and this distinction might persist. Even as it seems like a silly possibility, that’s not obviously a good heuristic for ruling it out. (It’s possible OpenAI adopted the Sol/Terra/Luna naming convention in part to make it more likely that any Mythos-like restrictions get appropriately scoped to Sol 5 models only, rather than to GPT-5.6+ as a whole.)

Vladimir_Nesov 26 Jun 2026 22:28 UTC
5 points
0
in reply to: Petropolitan’s comment on: Vladimir_Nesov’s Shortform
Dwarkesh Patel is unfortunately not insisting on the distinction between novel deep skills, and the stuff that in-context learning can handle (either simple skills or existing deep skill that are already in the weights). Because of this distinction, sample efficiency for novel deep skills can’t be usefully found in in-context learning, or in continual learning methods that repackage the capabilities of in-context learning using some sort of recurrent state that enables functionally unlimited contexts (memory/skill note directories explicitly maintained by the model also fit this description).

Some hypothetical continual learning methods of the future might be good enough for learning deep skills, better than the present day in-context learning is, but this is not a solved problem, it’s not something that can be expected to start working within some predictable amount of time. Dwarkesh Patel is caught up in essentially speculating how that could happen. One thing he mentions is on-policy self-distillation, which is an obvious enough idea to try, at least after you know about OPD itself, so every month it’s not demonstrated to be RLVR-level capable (when used outside RLVR) is evidence it’s not ready to soon become important for continual learning of novel deep skills (rather than for something else, like augmenting RLVR with process level feedback, maybe as a step in some recipe for RLVR training of multiple teacher models intended for eventual OPD into the target model).

Another thing he points to is “dreaming” that seems to be about model based RL (the example he gives is EfficientZero), but that has been a very long-running research direction, so the fact that it doesn’t work with LLMs yet is already significant evidence that it can’t be expected to start working within a short predictable amount of time. If you ignore all the technical distinctions, model based RL is somewhat similar to learning in self-constructed RL environments in RLVR, but focusing on RLVR specifically (rather than on some other RL shaped thing with constructed environments) is crucial because RLVR is the only thing that already works for teaching deep skills to LLMs. And RLVR is known to work well specifically when it happens as part of the overall model training/development/building process, not necessarily in long standalone sequences of post-deployment fine-tuning.

With current methods, sample efficiency in acquisition of deep skills can only be achieved by constructing the training data for RLVR (tasks/environments/graders), based on understanding the difficulties posed by the absence of specific novel deep skills (and the understanding of these difficulties can be obtained sample efficiently with in-context learning). This is impractical for human AI engineers to do manually for every little on-the-job thing, but can become practical once models can do that automatically on their own (which the remaining scaling runway likely allows).

Vladimir_Nesov 24 Jun 2026 20:07 UTC
12 points
0
on: Fable in Shackles
Non-US Non-Chinese AI companies need a currently-implausible level of funding to keep scaling at the frontier level, while the Chinese AI companies need large scale-up systems that aren’t being sold to them. It wouldn’t matter if they have enough FLOP/s in total in the form of servers that can’t fit large models, H100/H200/B200 (NVL8) rather than GB200/GB300 Oberon (NVL72) or future Kyber racks. It remains unclear when Huawei will actually start producing their large scale-up systems (CloudMatrix/Atlas) in important quantities, let alone in quantities that would match the frontier US compute, while maintaining a similar level of system-level performance (for individual scale-up systems needed to run large models).

So far, the Chinese models remain at a roughly 1T total params scale, which fits in the older 8-chip servers. But the 2025-2026 Oberon racks enable 20-25T total param models, then 2028 Kyber racks enable 400T total param models, and 8x Kyber Feynman systems of 2029-2030 enable 1-3 quadrillion param models. These models absolutely can’t run on the 8-chip servers currently used to run the 1T total param models, no matter how many such servers you have. My guess is that it’s actually useful to scale models almost as far as that hardware allows, and it’s not as expensive as it sounds once the large scale-up systems are available.

It is also unclear whether Chinese labs, or American open-source labs like Reflection and Arcee, can come up with algorithmic innovations to leapfrog the closed-source frontier.

You can’t sustainably leapfrog the closed-source frontier with algorithmic innovations, because the closed-source frontier will leapfrog you back with the same algorithmic innovations, only it’ll scale those innovations with more compute and to larger models, resulting in even higher quality.

Vladimir_Nesov 24 Jun 2026 17:01 UTC
7 points
0
in reply to: Petropolitan’s comment on: Model Size Scaling in 2023-2031
The Farseer paper does a lot of theoretical extrapolation from experiments with up to 3e21 FLOPs (and it really doesn’t like isoFLOPs, though the raw experimental data was published, so it’s possible to extract them). I expect compute optimal D/N ratio to be somewhat stable, merely on priors, and that is the sense in which I’m following Chinchilla scaling (the 240 tokens/param at 30x sparsity is quantitatively very different from Chinchilla’s 20 tokens/param). So I’m not engaging with the Farseer vs. Chinchilla debate that the Farseer paper is having, about the functional form of how loss or BPC depend on D and N, because I reject that framing and don’t reason from the considerations relevant there, even as my conclusions side with Chinchilla. Instead, I’m looking at whatever experimental data about the D/N ratio I can find, and running with it all the way to 2e29 FLOPs (modifying it with the split-the-shortfall rule once the data runs out, which I think is plausible with up to 4-5 epochs of repetition).

Arbitrarily high sparsity seems useful with unlimited data, see Figure 11 from this paper (the horizontal axis is log-sparsity; it looks like infinite sparsity gives an infinite compute multiplier). But sparsity drives the D/N ratio higher, seemingly in step with the compute multiplier. That is, 8x sparsity gives a 3x compute multiplier over dense, but also drives the compute optimal D/N ratio 3x higher compared to dense; and 30x sparsity gives a 6x compute multiplier, driving the D/N ratio 6x higher over dense (see Figure 12, left for the relevant isoFLOPs). Thus very high sparsity would ask for even more data, and so by the split-the-shortfall rule would need even more epochs of repetition (the 2031 model from this post already reaches 4 epochs). When data is scarce, high sparsity gets less helpful.

For the Chinese models, which target the 1T total params generation (pre-Oberon) scale-up systems for inference, it makes sense to fill all the available space above the active params up to about 1T total params with sparsity. But if they had more active params with the same number of total params, the quality would’ve been higher (as would the cost of pretraining and inference). So while keeping the total params count fixed, lower (rather than higher) sparsity gives better quality (and needs less data for a given amount of pretraining compute). It’s only when you keep the active params count (or pretraining compute) fixed, and the total params are not constrained, that higher sparity gives better quality. The reason to go for high sparsity (while observing the roughly 1T total params upper bound) is insufficient pretraining compute, or inability to bear the inference cost of models with more active params, and both these factors are plausibly why the Chinese models have so few active params and therefore high sparsity.
What links here?
- Vladimir_Nesov's comment on Reevaluating AI-2027: timelines, takeoff, alignment and China by StanislavKrym (6 Jul 2026 1:12 UTC; 8 points)
- Vladimir_Nesov's comment on Reevaluating AI-2027: timelines, takeoff, alignment and China by StanislavKrym (6 Jul 2026 2:41 UTC; 4 points)

Vladimir_Nesov 23 Jun 2026 3:41 UTC
12 points
0
in reply to: StanislavKrym’s comment on: Model Size Scaling in 2023-2031
Input token cost scales with active params, essentially doesn’t depend on total params (the same as the FLOPs of pretraining), and goes down over the years as FLOPs get cheaper. Output token cost is determined by KV cache per token times average context length. KV cache per token scales with model dimension, which scales with square root of active params or even slower (if more active params translates to more layers or more experts per layer), that is it scales significantly slower than the active params themselves do. Of course, output token cost can’t actually get below that of the input tokens because the output tokens are HBM bandwidth bound (for big models), so their cost (not necessarily price!) remains at least 2-3x higher, but with longer contexts they become even more expensive compared to the input tokens. So at a sufficient context length, the scaling trends play out if we keep the context length fixed and vary the active params of models, with the output token cost increasing slower than the input token cost as the number of active params increases.

Most tokens are input tokens, so their pricing matters more for the AI company. In principle, output tokens could be priced with zero gross margin, they just shouldn’t be generated at a loss. The cost of the output tokens is kept at a healthy 5-6x compared to input tokens by increasing the allowed context length.

So let’s say the 2026 model with 1.3T active params is literally Mythos 5 and it’s priced at $10 per 1M input tokens. That’s already with a gross margin that might be above 70%, since the 70% figure could probably anchor Opus 4.6 price, which is $5 per 1M input tokens, exactly 2x lower than Mythos 5 price, and Mythos 5 probably doesn’t have more than 2x the active params than Opus 4.6, since it probably didn’t use more than 4x the compute of Opus 4 in pretraining, and both plausibly use compute optimally sized numbers of active params (maybe Mythos 5 has more sparsity than Opus 4) without yet running out of unique training data (active params scale with the square root of pretraining compute). Which is to say, the $10 per 1M input tokens price of Mythos 5 probably has significant room for coming down even today.

The 240T total params model of 2028 has only 7.9T active params, 6x more than the 1.3T active params of the 2026 model, and it runs on Rubin instead of Blackwell. GB200 produces 5e15 FP4 FLOP/s per compute die, Rubin produces 17.5e15 FP4 FLOP/s per compute die, GB200 rack needs 140 kW, Rubin Oberon rack needs 225 kW. The cost per year of a 1 GW datacenter (not just the racks) might be going up 20%, so the cost of the datacenter time per compute die goes up maybe 90%, but the FP4 FLOP/s go up 3.5x, and so the FP4 FLOPs in Rubin are 1.8x cheaper than the FP4 FLOPs in GB200 (the 2028 model needs Kyber rather than Oberon, but let’s assume the cost of time per compute die isn’t too different). Thus the 6x difference in active params becomes a 3.3x difference in cost. The price for the 240T total params model of 2028 might be $33/$165 per 1M input/output tokens. (The output token price is adjusted as appropriate by increasing the allowed context length, if it would otherwise try to get below 5x the input token price. And this is plausibly still at a more than 70% gross margin, so could be cheaper actually.)
What links here?
- Vladimir_Nesov's comment on Fable in Shackles by ykevinzhang (24 Jun 2026 20:07 UTC; 12 points)
- Vladimir_Nesov's comment on Reevaluating AI-2027: timelines, takeoff, alignment and China by StanislavKrym (5 Jul 2026 4:36 UTC; 8 points)

Vladimir_Nesov 23 Jun 2026 2:16 UTC
8 points
0
in reply to: bhalstead’s comment on: Model Size Scaling in 2023-2031
First-party compute is distinct from serving inference via clouds (IaaS as opposed to TaaS compute). The quote you gave (at 11:30 in the podcast) discusses IaaS plus TaaS, while I talk about just IaaS (which is more relevant for R&D and training). In the same interview, Dylan Patel also says Anthropic will get to 2+ GW end of 2026 (at 1:12:10 in the podcast, the link I gave in the post for the 10 GW claims), which likely refers to IaaS only.

I’m basing the 3-4 GW figure on multiple less-specific clues and interpolation. For Anthropic, the general IT power of compute trend of about 3x per year, the claim of 2+ GW end of 2026 probably implies 1+ GW end of 2025, then there’s 10 GW end of 2027, 1 GW of TPUs in the 2026 buildout, more Trainium in 2026, so I think 2+ GW end of 2026 likely resolves to 3 GW. For OpenAI, there are contracts with Oracle for Blackwell (see the table in this section of the TPU post), which is 2025-2026 incremental compute, and compute at the end of 2025 is 2 GW. Plausibly OpenAI gets more than 4 GW end of 2026, but then it might rely on first-party compute to serve tokens more than Anthropic, and so use less of it for R&D and pretraining. Also, if OpenAI gets back to the same 10 GW at the end of 2027 as Anthropic, the possible difference over 2026 balances out, so it shouldn’t be strategically relevant for them when sizing pretraining runs.

Vladimir_Nesov 23 Jun 2026 1:40 UTC
4 points
0
in reply to: jsd’s comment on: Model Size Scaling in 2023-2031
A more precise value is 27.6T. It’s an upper bound, so I rounded down. It’s a difference of 2%, at this level I might use rounding inconsistently, and depending on context I sometimes keep just 1-2 significant digits.

Vladimir_Nesov 23 Jun 2026 1:28 UTC
13 points
0
in reply to: Lanthimum’s comment on: Model Size Scaling in 2023-2031
The compute scaling is much less significant than a 24x difference in model size would imply under the all-else-equal scaling assumptions, which is 24x squared. It’s instead going from 1.3e27 FLOPs (probably Mythos 5, possibly GPT-5.5 as well, or its base model will get there with more training in GPT-5.6) to 2e28 FLOPs, a 15x increase. Then there’s the change from 8x sparsity to 30x sparsity, which should give a 2x effective compute boost, to the total of 30x (3x boost from 8x sparsity compared to dense, 6x boost from 30x sparsity, see Figure 11 from this paper). But also the 2026 model has enough data, while the 2028 model is under a 4.5x data shortfall, so unclear how much of the effective compute boost survives. Maybe the 2028 model has 20x more effective compute in pretraining than the 2026 model.

On the other side of the 1.3e27 FLOPs, a 20x decrease gives 6e25 FLOPs, 3x less than late 2023 to very early 2024 compute that probably trained GPT-4o (though the model was likely significantly smaller than compute optimal). This same pretrain probably became o3 and then GPT-5.1 later in life. Perhaps the GPT-4o base model being smaller/overtrained cancels out the difference in compute, so it might be at the level of a compute optimal 8x sparsity model pretrained with 6e25 FLOPs. And GPT-5.1 was maybe around the level of Sonnet 4.5 if we try to ignore the differences in how they were post-trained, so the 2028 model might be stronger than the 2026 model by about as much as Mythos 5 is stronger than Sonnet 4.5 or GPT-5.1.

Then the 2031 model is using merely 11x more pretraining FLOPs than the 2028 model, and uses the same 30x sparsity (so there is no boost to effective compute from more sparsity), and has a 15x training data shortfall instead of 4x. So the difference between the 2028 and 2031 models is maybe 1.5x smaller than the difference between the 2026 and 2028 models (comparing the effective compute on the logarithmic scale), unless much larger pretrains have some currently-unknown advantages not captured in their token prediction loss. Perhaps the 2031 model is smarter than the 2028 model by about as much as Mythos 5 is smarter than the average of Sonnet 4.5 and Opus 4.6.

(Standard caveats, algorithmic advances can happen suddently at any time, any predictions about the current paradigm are only relevant as long as the current paradigm itself remains relevant.)
What links here?
- Vladimir_Nesov's comment on Reevaluating AI-2027: timelines, takeoff, alignment and China by StanislavKrym (5 Jul 2026 4:36 UTC; 8 points)

Model Size Scaling in 2023-2031

Vladimir_Nesov22 Jun 2026 23:07 UTC

85 points

16 comments12 min readLW link

Vladimir_Nesov

Model Size Scal­ing in 2023-2031

Model Size Scaling in 2023-2031