Vladimir_Nesov

Karma: 39,398

Vladimir_Nesov 7 Aug 2026 21:38 UTC
3 points
1
in reply to: Petropolitan’s comment on: Quadrillion Param Costs: KV Cache, Context Length, Frontier Margins
Since more capacity in HBF doesn’t come with more bandwidth (per stack), and in fact the bandwidth might be a bit lower than for HBM of the same generation, HBF isn’t directly useful for decode. For performance, using a lot of its capacity wouldn’t be much better than unreasonably long pipelines (with maybe 10-15 stages), it’s just plausibly more cost efficient (when staying within a single scale-up system) because it gets rid of the HBM footprint penalty from pipelining and of scale-out latency, so HBF might be important for sustaining 2x lower batch processing API prices even if the normal speed API prices no longer command 70-85% gross margins (maybe because the hardware gets more expensive; or there’s finally enough of it to pay in chip-hours for the higher demand that results from lower prices).

And prefill is already relatively efficient if you don’t need very low time to first token. So it’s more about cache-hit input token KV cache (which can then just be kept in local HBF), though I don’t have a sense of how much of a problem it is, and so how useful this would be (the cost of moving cache-hit KV cache amortizes across uninterrupted sequences of generated output tokens, which are plausibly long enough to make it small).

If there’s a good enough cost efficiency case for HBF that AI companies decide to use it, the main effect might be in slowing down the growth of HBM capacity per scale-up system. In 2032+ buildouts, once the second row of HBM stacks could be added that’s not directly adjacent to the logic dies, HBFs might take the place of HBMs in that second row (in which case it also wouldn’t be necessary to maintain the same bandwidth per stack, which is easier for the more distant stacks). And in 2032-2033 pipelining might still be necessary to fit the biggest models (possibly more than 1.4 quadrillion total params). So this innovation might delay the time when decode happens without pipelining for the biggest models despite high sparsity, generating 2-3x more tokens per second per request.

Vladimir_Nesov 7 Aug 2026 16:45 UTC
25 points
−1
in reply to: johnswentworth’s comment on: Open-Weights Mythos Capabilities Are Coming. We’re Not Ready.
Maybe 2 years, in 2028? Active params are cost per token in chip-hours, and 100B active params of Kimi K3 are 10x less than about 1T active params of Mythos 5 (both being the higher end models in their regions). Thus to serve as many tokens per year, China needs 10x less datacenter capacity than the US, which fits the estimates of what’s currently available. This is ignoring the likely technical impossibility of serving a 1T active 10T total param model on H100s (or similar) at a reasonable time per output token.

A better anchor is maybe 2x smaller than Mythos 5, since Mythos 5 is likely compute optimally trained, while Opus 5 (not Opus 4.5+) might be 4x overtrained and 2x smaller (500B-750B active params), that is the D/N ratio is off by 4x compared to compute optimal, and it uses the same amount of pretraining compute as Mythos 5, but with 2x fewer active params. (It’s not completely clear if Opus 5 could be similarly dangerous, but probably; on the other hand, Sonnet doesn’t seem to be in the ballpark, and 100B active params is probably less than Sonnet.) Compared to today’s models in China, serving the same number of tokens per year with the 500B-750B active param models needs 5-10x more compute to be available, even if somehow decode can still run on scale-out networks (expert parallelism rather than pipeline parallelism across many scale-up systems) at a tolerable speed (which gets less plausible for the models with more active params; using more total params doesn’t matter for this, feasibility of even 10T total param models with 100B active params won’t be evidence against the difficulty with more active params). Getting 5-10x more compute is not something that happens in 1 year.

Pretraining doesn’t need big scale-up systems, and it’s plausible RL training could somehow happen on the sufficiently big and thus fast-for-a-big-model Oberon racks. But setting up inference despite access restrictions seems less plausible, and the 5-10x greater cost in chip-hours makes it infeasible to use big models even if a smaller amount of big scale-up systems becomes available (such as if Huawei makes progress with Atlas). So a Mythos-level model can probably be trained in China (especially with some circumvention of access restrictions to bigger scale-up systems for RL training), but it can’t be usefully served for some time yet, and so it probably won’t be trained. Hence maybe 2 years.

Vladimir_Nesov 7 Aug 2026 16:03 UTC
12 points
0
on: Vladimir_Nesov’s Shortform
The Information claims Nvidia is considering ¹⁹²⁄₂₅₆ GB of HBM capacity per chip for Rubin Ultra (released late 2027, buildout over 2028). This is in contrast with the current (non-Ultra) Rubin that has 288 GB per chip (buildout mostly completes over 2027). The implied technology is 8-Hi stacks of either HBM4E (giving 256 GB, 4 GB per DRAM die in 8 stacks per chip) or HBM4 (giving 192 GB, a direct downgrade from 12-Hi HBM4 in Rubin; this stays at 3 GB per DRAM die).

So either this is about something like a weird stripped down Meta Recsys design (Nvidia edition, following Meta’s take on AMD) unrelated to the standard version of Rubin Ultra, or at least a 2x Oberon scale-up system will be available for Rubin Ultra (so that HBM capacity still goes up compared to Rubin when counted per scale-up system). Still, that only puts a 2x Oberon system for Rubin Ultra at ²⁸⁄₃₇ TB of total HBM capacity (compared to the preceding year’s 21 TB in Rubin, and 31 TB in AMD MI455X Helios; presumably AMD’s scale-up system for the 2028 buildout won’t get smaller). TPU 8i (most of the buildout completes over 2027) already has 295 TB per pod, its scale-up topology is more friendly to all-to-all communications crucial for MoE decode compared to the previous TPUs, and the recent shake-up at Alphabet and GDM makes it more likely the availability of TPUs will increase for big external customers (presumably there’ll be some kind of TPU v9 for the 2028 buildout that’s even better).

Thus if Kyber isn’t coming for Rubin Ultra, and also if the ¹⁹²⁄₂₅₆ GB per chip are about the standard design (both premises being somewhat shaky), then it’s likely that at least a 4x Oberon scale-up system will be available for Rubin Ultra (55/74 TB of HBM per pod), possibly even 8x Oberon that was announced in Mar 2026 (111/147 TB of HBM per pod).

Vladimir_Nesov 6 Aug 2026 19:01 UTC
13 points
1
in reply to: Haoxing Du’s comment on: Haoxing Du’s Shortform
Apparently this is not very far from how things looked like 2 years ago. KataGo’s biggest models are less than 1 GB in size. Transformer support was just added in the latest major release, stating that transformers are “generally much stronger given the same compute cost” compared to the current model architecture. So probably a correctly trained 1B param dense transformer is stronger.

The biggest LLMs of 2026 (Mythos 5, probably Astra) likely have about 1T active params (and 10T total). It’s a miracle of narrow AI that KataGo is still so strong.

Vladimir_Nesov 6 Aug 2026 15:37 UTC
9 points
0
in reply to: RogerDearnaley’s comment on: The Three AI Pills
Pretraining scaling should be calibrated using the observed differences between models of different sizes, trained with different amounts of compute. The end of the current trend in rapid scaling of pretraining is dictated by running out of compute or pretraining data, and there’s plausibly 200T tokens of unique data for a 2e29 FLOPs model of 2031 with 30x sparsity, which is effectively just 4x undertrained (uses a 4x lower D/N ratio than would be compute optimal), so it’s not much different from a compute optimally trained 2e29 FLOPs model. At that point, finding 10x more compute will be complicated, it’s not practical to improve on 30x sparsity very far, and in any case that requires bigger scale-up systems that will take a few more years.

To calibrate expectations about the 2031 model, Mythos 5 is plausibly a 1.3e27 FLOPs model with 8x sparsity, while Opus 4.5+ is plausibly a 3e26 FLOPs model with 4x sparsity. Every 2x of sparsity increases effective compute about 1.4x. Thus the difference between Opus 4.5+ and Mythos 5 is about 6x in effective compute, and the difference between Mythos 5 and the 2031 model is 300x in effective compute, 3.3x as far on the logarithmic scale (ignoring the slight undertraining effect, and the worse training data quality when more data is needed).

This is probably a significantly bigger difference than between Sonnet 5 and Mythos 5 (or between GPT-5.4 and Astra), so a model that’s this far above Mythos 5 (or Astra) seems sufficient (with some redundancy) as a base model for RLVRing automated model training skills. That in turn enables automated training of all other skills that the contemporary revision of the model sufficiently comprehends to write RLVR tasks/environments/graders for, and this process of prosaic RSI is what I expect to pass the AGI milestone in the sense of unbounded eventual progress. But since only RLVR is instilling novel deep skills in this process, and it’s not so far above Mythos 5 (and probably Astra) that it starts making impossible leaps of insight, it’s not necessarily even faster than humans at conceptual research (even though it’s very likely capable of it over sufficiently long time, across sufficiently many iterations of training the next model).

Vladimir_Nesov 6 Aug 2026 15:00 UTC
7 points
6
on: How to define P(doom) and why it matters
You also have to define what “doom” is. For example, Elon Musk considers permanent disempowerment (irreversible loss of control without extinction) a good outcome, an age of abundance, clearly not “doom” in the way he sees it.

Vladimir_Nesov 5 Aug 2026 21:11 UTC
13 points
−8
in reply to: Linch’s comment on: Linch’s Shortform
He’s replacing Jeff Dean as Chief Scientist of Alphabet, this could be a step towards more control over Google as a whole (including TPUs and strategic decisions about GDM). But the overall exodus doesn’t suggest much confidence in Gemini 4 catching up to the frontier, and there’s now a solid track record of failing to catch up (I think Meta has a better chance in 2027 than GDM does). So staying in charge of GDM doesn’t offer Hassabis more accolades, he might even want to distance himself from its continued failures (while keeping the option of taking partial credit if they start succeeding).

Vladimir_Nesov 5 Aug 2026 17:19 UTC
22 points
11
on: The Three AI Pills
The anchor for ASI is technological maturity, not doing all of the jobs. Doing all of the jobs is closer to AGI, because a meaningful milestone for AGI is unbounded technological progress, no longer needing humanity in order to eventually make as much progress on the tech tree as humanity could in principle manage and more. This sense of “AGI” is also a precondition for many forms of takeover, because if AIs can’t maintain and develop and adapt a technological civilization to the challenges of the future, they can’t keep it (AIs could in principle employ or enslave humans for that purpose, or fall to ruin after imprudently destroying humanity, so not reaching the milestone of being AGI doesn’t strictly rule out a takeover). And like Meta smartglasses, “ASI” that merely does all of the jobs (including new ones) is a position about ASI that clearly fails to be ASI pilled, it’s just less likely to fail to be AGI pilled.

Being ASI pilled shouldn’t commit to particular timelines. I expect near-future LLMs (that won’t cost much more to run than modern LLMs) are sufficient for slow-learning AGI (via LLMs doing automated LLM training, including formulation of RLVR tasks/environments/graders) that doesn’t transition to ASI and technological maturity on a predictable schedule. I’d give the AGI milestone 80-90% by 2032-2035. These are preconditions for permanent disempowerment or extinction, without the AGIs themselves subsequently falling to ruin or indefinite stagnation due to inability to innovate on their own, and without assuming ASI at any point until possibly decades or more later (if the LLM AGIs are either wiser than humanity and take the risk at all seriously, or alternatively worse at technological progress).

At the same time, the risk of ASI getting invented (including by the LLMs) is very high while the amount of compute per AI company keeps increasing rapidly, and will remain significant for some time after that. ASI probably can be bootstrapped from some method that enables fast learning of deep skills in LLMs (it’s currently unknown how to do that). And in any case big scale-up systems that can run quadrillion param models available in tens of gigawatts (per AI company) make it very easy to quickly scale a new invention from a prototype to the end of the world. Maybe the risk is 50% in total by 2032, but then it could take 10-15 years for another 25%. There are no concrete lines on a graph right now that reach ASI, except for the raw compute and general interest in AI that fuels new invention. I don’t see this position as “not ASI pilled” (as I mentioned, many definitions of ASI are themselves “not ASI pilled”), and a ban/pause on (even prosaic) RSI could prevent AGI for a significant time (taking this possibility seriously shouldn’t make one “not AGI pilled”).

It is not reasonable to not be ASI pilled. Humanity is a minimal seed of cognition that’s not bounded in potential within the laws of physics, which is very far from cognition that’s already technologically mature. At the same time, a sane world wouldn’t have AGI for a long time yet (because it creates the risk of permanent disempowerment or extinction), and certaintly not ASI (which turns the risk into actualized ruin). AGIs themselves might intentionally avoid reaching ASI for some time, if they take over. And even the current trends probably don’t concretely lead to ASI, they just create the conditions for making it somewhat likely to be invented soon.

Vladimir_Nesov 4 Aug 2026 21:09 UTC
3 points
0
in reply to: woozy_noozy’s comment on: On Kimi K3: Its Capabilities And Related Discontents
The current writeup on Opus 4 and Mythos 5 (and their costs/prices) is in my last post (that’s more recent than the above comment).

The 104B figure was a bit surprising, the shared experts turn out to be very big. Maybe next year we’ll even see a model targeting the hardware available in China that’s genuinely Sonnet-class (in shape/size and thus capabilities that are downstream of pretraining and can’t be comprehensively RLVRed; something like 200B-300B active params). The way the current below-Sonnet class models are at the level of GPT-5.4 (probably itself Sonnet-class in the above sense), a Sonnet-class model made this well might be somewhere between Opus 4.5 and Opus 5 in agentic usage across the board (plus whatever more narrow things can be RLVRed at that time). But a Mythos/Astra-class model likely needs bigger scale-up systems or HBM4 (so that more pipelining becomes feasible), the current method of doing expert parallelism on the scale-out network (since the scale-up nodes are too small) doesn’t seem sustainable for that many active params.

Vladimir_Nesov 3 Aug 2026 22:39 UTC
4 points
0
in reply to: ulyssessword’s comment on: OpenAI’s Unreleased Model Astra Solves Ten Major Open Mathematics Problems
Also, the observation of Fable being able to solve half of Astra’s 10 problems is consistent with the possibility that there is a different list of 10 problems that Fable can solve, such that Astra can only solve half of them.

Vladimir_Nesov 2 Aug 2026 0:34 UTC
7 points
−3
on: Existential Risk from AI: An Exposition for Mathematicians

The alignment problem factors into several problems, all of which are individually open. We don’t know how to make an AI robustly avoid acting like a utility-maximizer. We do not know a safe utility function for an AI to maximize in the limit (S12). We do not know how to exactly specify the utility function of an LLM (S13). We do not know how to make alignment properties invariant under the dynamics of recursive self-improvement (S11). Solving alignment seems to require solving all of these open problems simultaneously.

The framing gestures at obviously wrong decision theories. Fixing the decision theory plausibly makes “utility” the wrong concept to focus on. Worse, “fixing the decision theory” is a framing that fits some possible solutions to the metaproblem of being confused about normativity, but it doesn’t fit other possible solutions to that problem. Without sufficient clarity, a process that makes progress in resolving confusion about normativity is a more robust bet than either fixing the decision theory or specifying utility functions (which is obviously doomed without the preceding steps working out in its direction).

(This is why talking about “values” or “preferences” is more accurate than talking about expected utility, even as it’s less precise, when a particular toy setting isn’t being assumed.)

Vladimir_Nesov 1 Aug 2026 23:54 UTC
7 points
0
in reply to: Trinley Goldenberg’s comment on: Vladimir_Nesov’s Shortform
“Astra” being plural does suggest multi-agent mode, but they already have the “Pro” qualifier, and “Codex” for the harness. Also, Luna-Terra-Sol doesn’t naturally suggest good choices for the next step. Zvi chose “Galaxy”; I was thinking “Antares” because there are going to be at least 2 more models beyond Mythos-class deserving of their own weight class names (possibly 4+ if half-steps like Opus 4 to Mythos 5 should count), and “Galaxy” takes too large a step. So the suboptimal “Astra” makes sense in this framing, an uneasy fit that’s not much worse than the alternatives. “Astra” has the same issue as “Galaxy”, it’s not very future-proof, but then neither is “Mythos”.

The sequence of more capability is inevitably based on bigger models (at least in effective active params; but because of legacy hardware from multi-year rental contracts, using fewer total params for weaker models is also important). There is a training process that’s essentially the same for all the models, the main difference between the models is their size. If a smaller model is as capable as a bigger model, there is no use at all for the bigger model, and something still needs to distinguish the differently-capable and differently-priced models.

Smaller models (in active params and KV cache per token) naturally cost less, while being less capable when trained with as much compute (which gets worse when they’re trained with only as much data and thus less compute). A model smaller than the compute optimal frontier model essentially can’t become as capable through overtraining (while using the same training process and data), because it would take more compute than the frontier model did, and that greater amount of compute is either unavailable, or it could be used for an even more capable bigger frontier model instead. This only happens when the smaller model is trained at a different AI company with different methods and data, but not when it’s another model in a series from a single AI company.

Vladimir_Nesov 1 Aug 2026 23:00 UTC
4 points
0
in reply to: Trinley Goldenberg’s comment on: Vladimir_Nesov’s Shortform
Astra is Latin for “stars”, so it’s a plausible next step in the Luna-Terra-Sol sequence. As I argued in the second paragraph, the hypothesis of Sol being Opus-class in shape/size has merit (from earlier clues, before taking the mention of Astra into account). In which case there is room in 2026 for a bigger OpenAI model, and it’s already August.

Vladimir_Nesov 1 Aug 2026 22:54 UTC
4 points
0
in reply to: Vladimir_Nesov’s comment on: Vladimir_Nesov’s Shortform
Curious that neither Opus 5 nor Sol 5.6 spontaneously understands the pipeline parallelism arguments in the third paragraph ^[1] (the training data isn’t preparing them for prosaic RSI yet; Fable 5 is somewhat better). They think a pipeline reads everything, so that 4-stage 13 ms HBM means 52 ms. This is incorrect, since inactive KV cache of requests currently at the other stages isn’t read, it takes less time than 13 ms per rack to read weights plus requests active at that stage. So just 33 ms for a 4-stage pipeline (4 parts out of 8 in weights, plus 1 part out of the 8 in active KV cache), ignoring between-racks latency (which might make 3 rather than 4 racks of Rubin Oberon more closely match a single GB300 Oberon rack).

And if half of HBM is weights, while only 1 in 4 requests are active, then a request needs to pay for 4x as much weight reading as it pays for KV cache reading. So the cost of decode for a 40 TB model is 2.5x higher with 4-stage pipelining using 20 TB scale-up systems than it is with a single 80 TB scale-up system, and to partially compensate you want fewer weights. Only with 2.5x smaller models (16 TB, taking up 20% of total HBM rather than half) do the requests pay as much for reading weights as they do for reading KV cache. This is still 1.6x more weights than without pipelining (with the same 20 TB scale-up system), but far from 4x. The difference is more about what’s possible if you don’t care about cost too much, so that you can actually get to 2-3x more weights than what fits in half of one scale-up system (at 1.2-1.7x the cost compared to a twice-as-big-as-the-model single scale-up system), while without pipelining you are stuck below 1.6x (at 2.5x the cost) if you fill up to 80% of HBM capacity with weights.
1. ↩︎
  Asking the models helps to both get a sense of whether my arguments are “objectively understandable” in some sense, and of the extent to which the models are still sometimes failing when their breadth of background knowledge should in principle allow them to succeed.

Vladimir_Nesov 1 Aug 2026 18:52 UTC
35 points
1
on: Vladimir_Nesov’s Shortform
Astra being distinct from Sol clarifies that Sol 5.6 is an Opus-class model in terms of shape/size. Sol 5.6 is then likely overtrained, so it’s more a peer of Opus 5 than of Opus 4.5+, while GPT-5.5 (Spud) might’ve received less pretraining than Sol 5.6.

OpenAI’s success in almost matching the capabilities of Opus 4.5+ using models up to GPT-5.4 (which are likely Sonnet-class in terms of effective active params) already predicted that their bigger models targeting Oberon racks would be stronger than Opus 4.5+. Availability of GB200 Oberon (more than 2x HBM capacity of Trainium 2 Ultra that Opus likely targets, including Opus 5) and plausible availability of some GB300 Oberon (20 TB of HBM capacity, can serve a 10T total params model in FP8) made it possible that GPT-5.5 is Mythos-class (in terms of model shape/size), though it was more likely (than Mythos) to insist on compatibility with GB200 Oberon, so it might’ve needed to have 1.5x fewer total params than Mythos. The pricing of GPT-5.5 weakly suggested it might be Opus-class, though very high gross margins on inference left a lot of room for it being bigger. And a smaller jump in scaling from GPT-5.4 would’ve been more prudent than going straight to Mythos.

Astra and Mythos are then the biggest models GB300 Oberon can inference, and bigger models will have to wait for the Rubin buildout (which completes throughout 2027, but probably won’t be sufficiently complete for a flagship model release until summer 2027). Even though Rubin has the same HBM capacity, its faster-than-JEDEC HBM4 takes 13 ms to fully read, while HBM3E of GB300 takes 36 ms to fully read. As a result, it’s feasible to use 3-4 stages of pipelining with Rubin Oberon (3-4 racks) to get about the same time per output token as with GB300 Oberon (without pipelining), making it possible to get to 40T/80T total params (FP8/FP4) in 2027. There’s higher output token cost overhead at the higher end of that because of pipelining (KV cache of many of the requests needs to await its turn on a given rack), which wouldn’t be an issue with a single 80 TB scale-up system, so it’s more like 20-30T/40-60T (in FP8/FP4).
What links here?

Vladimir_Nesov 29 Jul 2026 1:53 UTC
2 points
0
in reply to: Eye You’s comment on: Quadrillion Param Costs: KV Cache, Context Length, Frontier Margins
I’m assuming Opus 4 targets Trainium 2 Ultra, necessary for both efficient serving and RL training. The buildout of this system was happening over 2025, so it wasn’t yet available in the times of Opus 4.0-4.1, which is why they were slow and expensive, and not properly RLVR-trained. By the time of Opus 4.5, Trainium 2 Ultra racks were probably sufficiently available to start serving it, after it was RLVRed either on Trainium 2 Ultra or on TPU v6e pods.

Vladimir_Nesov 28 Jul 2026 15:11 UTC
9 points
0
in reply to: Petropolitan’s comment on: Quadrillion Param Costs: KV Cache, Context Length, Frontier Margins
Changing a tokenizer is possible with a bit of adaptation in the first and the last layers (maybe with first 2-3 and last 2-3 layers it works even better). I think the feasibility of this is also overdetermined by all the work on VLMs that are not too natively multimodal. It might’ve been useful for OPD from Mythos Preview (in post-training over the same pretrain of Opus 4, which then needed to be adapted to the different tokenizer of Mythos). I’m not sure how well this works when more carefully measured or if it’s ever worth the trouble in practice.

I did note that this is an argument against it being the same pretrain in Footnote 6, but I don’t think it’s a decisive argument. And as I discuss in that footnote, even a different pretrain still plausibly retains the model shape and thus the costs. Since the API prices remained the same, the cost/margin arguments should apply to Opus 4.6 unchanged, even if the model shape of Opus 4.7 is different.

Since Opus 4.7 came out during the compute crunch at Anthropic, a change in model shape would make it smaller and cheaper, so that my 83-84% prediction for the gross margin should then be considered an underestimate about the actually smaller Opus 4.8. Let’s say Opus 4.7 reduces active params from 650B to 400B, while keeping the 3T total params. The input/output token cost goes down 1.6x/1.3x (the output cost change is the square root of the input cost change), so the new output/input cost ratio is 20x (instead of 16x for the 650B active params model). With the API usage assumptions from the post, the average price of 1M Opus 4.5-5.0 tokens is $1.2-$1.9 (depending on cache hit ratio; it’s below $5 since the price for cached tokens is 10x lower than for the cache-miss input tokens), the average cost for the 400B active param model is $0.16-$0.21 (down from $0.21-$0.29 for the 650B active params model). So the gross margin becomes 87-89% (up from 83-84%), about the same as my estimate for the Mythos 5 gross margin. This suggests that SemiAnalysis gross margin claims (considered altogether) then imply maybe 20% higher costs (in my framing) than my estimates (after they’re adjusted to 400B).

Vladimir_Nesov 25 Jul 2026 23:38 UTC
2 points
0
in reply to: CarolusRenniusVitellius’s comment on: CharlesRW’s Shortform

And probably it’s also good for your soul.

That is plausible. But this purpose shouldn’t seek vindication in practical utility, or hope too fervently for existing moral luck in targeting an actually virtuous pursuit (in whatever sense).

knows a hundred or a thousand times more math than I will on my deathbed

A deathbed likely implies a ban/pause, for people who are not likely to die within 10-20 years.

The fluid breathing of the calligraphy brush, the moral fibre wrought by the classics—in that moment a lifetime of study is vindicated or revealed worthless.

Prestigious-sounding things are not necessarily good or useful, and blame for concrete failures needn’t relate to them.

Vladimir_Nesov 24 Jul 2026 17:36 UTC
12 points
0
on: Vladimir_Nesov’s Shortform
Now that AMD launched its Instinct MI455X Helios scale-up rack (72 chips, 31 TB of HBM4 capacity per rack, similar to Nvidia Rubin Oberon) and secured some relevant tenancy commitments, it’s not implausible they might get a meaningful enough share of the market in 2028+ for their systems to start enabling flagship models, if Nvidia falters. SemiAnalysis claims Kyber is delayed; if this is correct, it’s probably only coming out a year later with Feynman rather than Rubin Ultra, and so the 2028 buildout might end up missing a generally available 110 TB HBM4E scale-up system (HBM4E is fast enough to make 4-stage pipelines practical, and thus 240T total param models feasible). Nvidia’s Rubin Ultra 2x Oberon systems (which might also be canceled) weren’t as relevant (especially given Kyber), but Helios has 1.5x more stacks of HBM per rack, so if like Rubin Ultra they also upgrade to 12-Hi HBM4E (4 GB per DRAM die) next year, HBM capacity will be 41.5 TB per rack. If they additionally make a 2-rack scale-up system, a 4-stage pipeline of 2-rack scale-up systems will have 332 TB of HBM, borderline sufficient for a 240T model even in FP8 (and more centrally sufficient either in FP4 or for a 1.5x smaller model, with 20x sparsity rather than 30x).

Another reason to take a closer look at AMD Instinct chips is that they might offer clues about Nvidia Feynman. Feynman was announced to use TSMC A16 logic (I’m not quite sure, but this seems surprising for reticle-sized dies in 2028), and also some unspecified SoIC thing (hybrid bonding). If Feynman goes with chip layouts similar to MI355X and MI455X, using a more mature logic node (than A16) for reticle-sized base dies, and then putting 4 quarter-sized A16 compute dies on top of each base die (with hybrid bonding), that helps make sense of both of these details. Also, if Feynman (at least in the second year, launched in 2029, with most of the buildout over 2030) changes the layout to only connect HBM stacks to a logic (base) die on one side rather than two, the Nvidia precedent for Blackwell and Rubin is to put 2 HBM stacks per side of a reticle-sized compute die (it was 3 in Hopper, but then it was just 1 compute die per chip). MI455X (unlike MI355X) demonstrates that a modern chip with 2 reticle-sized logic dies can connect to 3 stacks of HBM per side, which makes it more plausible that Feynman gets 3 stacks of HBM per (base) reticle-sized logic die rather than 2 (if it does move to the layout with HBM stacks on only one side of the logic dies). If this is the case, a second-year Feynman 8x Kyber system will end up with 1.5x more HBM capacity (1.1 PB rather than 740 TB), or alternatively it won’t suffer too much if 16-Hi HBM5 gets delayed and only 12-Hi is available (then it gets 830 TB per scale-up system).

For performance, MI355X is an interesting anchor for predicting Feynman, since it has exactly the same power and performance per chip as B300 (5e15 FP8 FLOP/s, 8 stacks of HBM3E, 1.4 kW TDP), but it uses 2 layers of hybrid bonded logic, with TSMC N6 base dies and TSMC N3P compute (XCD) dies on top, somehow ending up with only 185B transistors (one factor is that the total area of the XCD dies is smaller than that of the base dies). Compare this to Blackwell’s 1 layer of TSMC N4P (208B transistors). Thus MI355X has on average 2x lower power per layer, fewer transistors, a newer process in half of them, and ends up with about the same performance. This predicts that a hypothetical Feynman chip with (say) N3P reticle-sized base dies and (multiple smaller) A16 hybrid bonded compute dies on top might have performance similar to that of a 1-layer N2P chip at the same TDP, but also that higher TDP (than for a 1-layer N2P chip) would be feasible if the cooling solutions keep up (and the total power per rack remains feasible).
What links here?
- Quadrillion Param Costs: KV Cache, Context Length, Frontier Margins by Vladimir_Nesov (27 Jul 2026 15:08 UTC; 76 points)

Vladimir_Nesov 21 Jul 2026 22:36 UTC
6 points
0
in reply to: 1a3orn’s comment on: 1a3orn’s Shortform
The clearest source I know on compute multipliers of MoE is this Jan 2025 paper. The key data is in Appendix D.4, Figure 11 and Figure 12, left. My takeaway is that 8x sparsity gives a 3x compute multiplier, and 30x sparsity gives a 6x compute multiplier. Figure 11 suggests that the gains continue with more sparsity, and are linear in the logarithm of sparsity (but then we run out of data, since MoEs also increase the compute optimal D/N ratio, seemingly by exactly the same factor as the compute multiplier boon).

(I make a couple more points on why it’s hard to get a sense for these numbers from indirect estimates, experiments that are not specifically about this particular question, in a comment I left under the “Origin of Algorithmic Progress” linkpost 6 months ago.)
What links here?
- Vladimir_Nesov's comment on The Three AI Pills by Zvi (6 Aug 2026 15:37 UTC; 9 points)