Vladimir_Nesov

Karma: 36,293

Vladimir_Nesov 18 Dec 2025 3:57 UTC
2 points
0
in reply to: tslarm’s comment on: A friction in my dealings with friends who have not yet bought into the reality of AI risk
Maybe there are modes of engagement that should be avoided, and many ideas/worldviews themselves are not worth engaging with (though neglectedness in your own personal understanding is a reason to seek them out). But as long as you have allocated time to something, even largely as a result of external circumstances, doing a superficial and half-hearted job of it is a waste. It certainly shouldn’t be the intent from the outset, as in the quote I was replying to.

Vladimir_Nesov 18 Dec 2025 3:20 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Cole Wyeth’s Shortform
If AGI is human-equivalent for the purposes of developing a civilization, a collective of AGIs is at least as capable as humanity, plus it has AI advantages, so it’s much more capable than a single AGI instance, or any single human. This leads to ASI being often used synonymously with AGI lately (via individual vs. collective conflation). Such use of “ASI” might free up “AGI” for something closer to its original meaning, which didn’t carry the implication of human-equivalence. But this setup leaves the qualitatively-more-capable-than-humanity bucket without a label, that’s important for gesturing at AI danger.

I think the other extreme for meaning of “ASI”, being qualitatively much stronger than humanity, can be made more specific by having “ASI” refer to the level of capabilities that follows software-only singularity (under the assumption that it does advance capabilities a lot). This way, it’s neither literal technological maturity of hitting the limits of physical law, nor merely a collective of jugged-human-level AGI instances wielding their AI advantages. Maybe “RSI” is a more stable label for this, as in Superintelligence Strategy framing where “intelligence recursion” is the central destabilization bogeyman, rather than any given level of capabilities on its own.

Vladimir_Nesov 18 Dec 2025 2:40 UTC
11 points
0
in reply to: JennaS’s comment on: A friction in my dealings with friends who have not yet bought into the reality of AI risk

you sympathize with them while not taking their worldview seriously

There is no reason at all to take any idea/worldview less than seriously. For the duration of engagement, be it 30 seconds as a topic comes up, or 30 minutes of a conversation, you can study anything in earnest. Better understanding, especially of the framing (which concerns are salient, how literal words translate into the issues they implicitly gesture at), doesn’t imply your beliefs or attitudes must shift as well.

if you aren’t willing to change your beliefs, why should they

This is not just an inadvisable or invalid principle, but with the epistemic sense of “belief” it’s essentially impossible to act this way. Beliefs explain and reflect reality, anything else is not a belief, so if you are changing your beliefs for any reason at all that is not about explaining and reflecting reality, they cease being beliefs in the epistemic sense, and become mental phenomena of some other nature.

Vladimir_Nesov 16 Dec 2025 21:03 UTC
5 points
−1
on: A friction in my dealings with friends who have not yet bought into the reality of AI risk

I will repeatedly interject something along the lines of “you keep talking about this as a problem that it falls upon me to solve, while in reality we are all sitting in the same boat with respect to existential AI risk, so that you in fact have as much reason as me to try to work towards a solution where we are not all murdered by superintelligent AIs a few years down the road”.

This demands that others agree with you, for reasons that shouldn’t compel them to agree with you (in this sentence, rhetoric alone). They don’t agree, that’s the current situation. Appealing to “in reality we are all sitting in the same boat” and “you in fact have as much reason as me to try to work towards a solution” should inform them that you are ignoring their point of view on what facts hold in reality, which breaks the conversation.

It would be productive to take claims like this as premises and discuss the consequences (to distinguish x-risk-in-the-mind from x-risk-in-reality). But taking disbelieved premises seriously and running with them (for non-technical topics) is not a widespread skill you can expect to often encounter in the wild, without perhaps cultivating it in your acquaintances.

Vladimir_Nesov 15 Dec 2025 16:28 UTC
3 points
0
on: AI in 2025: gestalt

Nesov notes that making use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM on inference chips, as is doing RL on bigger models. He expects it won’t be possible to do the next huge pretraining jump (to ~30T active) until ~2029.

HBM per chip doesn’t matter, it’s HBM per scale-up world that does. A scale-up world is a collection of chips with sufficiently good networking between them that can be used to setup inference for large models with good utilization of the chips. For H100/H200/B200, a scale-up world is 8 chips (1 server; there are typically 4 servers per rack), for GB200/GB300 NVL72, a scale-up world is 72 chips (1 rack, 140 kW), and for Rubin Ultra NVL576, a scale-up world is 144 chips (also 1 rack, but 600 kW).

use of bigger models (i.e. 4T active parameters) is heavily bottlenecked on the HBM

Models don’t need to fit into a single scale-up world (using a few should be fine), also KV cache wants at least as much memory as the model. So you are only in trouble once the model is much larger than a scale-up world, in which case you’ll need so many scale-up worlds that you’ll be effectively using the scale-out network for scaling up, which will likely degrade performance and make inference more expensive (compared to the magical hypothetical with larger scale-up worlds, which aren’t necessarily available, so this might still be the way to go). And this is about total params, not active params. Though active params indirectly determine the size of KV cache per user.

He expects it won’t be possible to do the next huge pretraining jump (to ~30T active) until ~2029.

Nvidia’s GPUs probably won’t be able to efficiently inference models with 30T total params (rather than active) until about 2029 (maybe late 2028), when enough of Rubin Ultra NVL576 is built. But gigawatts of Ironwood TPUs are being built in 2026, including for Anthropic, and these TPUs will be able to serve inference for such models (for large user bases) in late 2026 to early 2027.

Vladimir_Nesov 15 Dec 2025 16:05 UTC
4 points
0
in reply to: Roman Malov’s comment on: Roman Malov’s Shortform
The general principle is that sufficiently smart people by default win most competitions among 100 randos (given sufficient training, when that’s at all relevant) that they care to enter.

Vladimir_Nesov 14 Dec 2025 0:56 UTC
3 points
0
in reply to: Caleb Biddulph’s comment on: CBiddulph’s Shortform
To be “not-insane”, you don’t need rationality in this narrow sense, in most circumstances. You don’t need to seek out better methods for getting things right, you just need some good-enough methods. A bit of epistemic luck could easily get you there, no need for rationality.

So the issue of behaving/thinking in an “insane” way is not centrally about lack of rationality, rationality or irrationality are not particularly relevant to the issue. Rationality would help, but there are many more things that would also help, some of them much more practical for any given object level issue. And once it’s resolved, it’s not at all necessary that the attitude of aspiring to rationality was attained, that any further seeking out of better methods/processes will be taking place.

Vladimir_Nesov 13 Dec 2025 23:45 UTC
4 points
0
in reply to: Caleb Biddulph’s comment on: CBiddulph’s Shortform
Rationality is not correctness, not truth or effectiveness, it’s more narrow, disposition towards better methods/processes that help with attaining truth or effectiveness. Keeping intended meaning narrow when manipulating a vague concept helps with developing it further; inflation of meaning to cover ever more possibilities makes a word somewhat useless, and accessing the concept becomes less convenient.

Vladimir_Nesov 13 Dec 2025 3:18 UTC
2 points
0
in reply to: Spective’s comment on: Does dissolving newcomb’s paradox matter?
If Omega tells you what you’ll do, you can still do whatever. If you do something different, this by construction refutes the existence of the current situation where Omega made a correct prediction and communicated it correctly (your decision can determine whether the current situation is actual or counterfactual). You are in no way constrained by existence of a prediction, or by having observed what this prediction is. Instead, it’s Omega that is constrained by what your behavior is, it must obey your actions in its predictions about them. See also Transparent Newcomb’s Problem.

This is clearer when you think of yourself (or of an agent) as an abstract computation rather than a physical thing, a process formally specified by a program rather than a physical computer running it. You can’t change what an abstract computation does by damaging physical computers, so in any confrontation between unbounded authority and an abstract computation, the abstract computation is having the final word. You can only convince an abstract computation to behave in some way according to its own nature and algorithm, and external constructions aren’t going to be universally compelling to abstract algorithms (such as Omega being omniscient, or the thought experiment being set up in a certain way).

Vladimir_Nesov 10 Dec 2025 5:34 UTC
2 points
0
in reply to: faul_sname’s comment on: faul_sname’s Shortform
When you go through a textbook, there are confusions you can notice but not yet immediately resolve, and these could plausibly become RLVR tasks. To choose and formulate some puzzle as an RLVR task, the AI would need to already understand the context of that puzzle, but then training on that task makes it ready to understand more. Setting priorities for learning seems like a general skill that adapts to various situations as you learn to understand them better. As with human learning, the ordering from more familiar lessons to deeper expertise would happen naturally for AI instances as they engage in active learning about their situations.

I think the schleppy path of “learn skills by intentionally training on those specific skills” will be the main way AIs get better in the next few years.

So my point is that automating just this thing might be sufficient, and the perception of its schleppiness is exactly the claim of its generalizability. You need expertise sufficient to choose and formulate the puzzles, not yet sufficient to solve them, and this generation-verification gap keeps moving the frontier of understanding forward, step by step, but potentially indefinitely.

Vladimir_Nesov 10 Dec 2025 5:20 UTC
33 points
12
in reply to: Cole Wyeth’s comment on: Cole Wyeth’s Shortform
AI danger is not about AI, it’s about governance. A sane civilization would be able to robustly defer and then navigate AI danger when it’s ready. AI is destabilizing, and while aligned AI (in a broad sense) is potentially a building block for a competent/aligned civilization (including human civilization), that’s only if it’s shaped/deployed in a competent/aligned way. Uploads are destabilizing in a way similar to AI (since they can be copied and scaled), even though they by construction ensure some baseline of alignment.

Intelligence amplification for biological humans (that can’t be copied) seems like the only straightforward concrete plan that’s not inherently destabilizing. But without highly speculative too-fast methods it needs AI danger to be deferred for a very long time, with a ban/pause that achieves escape velocity (getting stronger rather than weaker over time, for example by heavily restricting semi manufacturing capabilities). This way, there is hope for a civilization that eventually gets sufficiently competent to navigate AI danger, but the premise of a civilization sufficiently competent to defer AI danger indefinitely is damning.

Vladimir_Nesov 10 Dec 2025 4:14 UTC
2 points
0
in reply to: artifex0’s comment on: Little Echo

if your effort to constrain your future self on day one does fail, I don’t think there’s a reasonable decision theory that would argue you should reject the money anyway

That’s one of the things motivating UDT. On day two, you still ask what global policy you should follow (that in particular encompasses your actions in the past, and in the counterfactuals relative to what you actually observe in the current situation). Then you see where/when you actually are, what you actually observe, and enact what the best policy says you do in the current situation. You don’t constrain yourself on day one, but still enact the global policy on day two.

I think coordination problems are a lot like that. They reward you for adopting preferences genuinely at odds with those you may have later on.

Adopting preferences is a lot like enacting a policy, but when enacting a policy you don’t need to adopt preferences, a policy is something external, an algorithmic action (instead of choosing Cooperate, you choose to follow some algorithm that decides what to do, even if that algorithm gets no further input). Contracts in the usual sense act like that, assurance contracts is an example where you are explicitly establishing coordination. You can judge an algorithmic action like you judge an explicit action, but there are more algorithmic actions than there are explicit actions, and algorithmic actions taken by you and your opponents can themselves reason about each other, which enables coordination.

Vladimir_Nesov 10 Dec 2025 0:59 UTC
2 points
0
in reply to: faul_sname’s comment on: faul_sname’s Shortform
AI currently lacks some crucial faculties, most obviously continual learning and higher sample efficiency (possibly merely as a measure of how well continual learning works). And these things plausibly fall under the umbrella of the more schleppy kinds of automated AI R&D, so that if the AIs learn the narrow skills such as setting up appropriate RL environments (capturing lessons/puzzles from personal experiences of AI instances) and debugging of training issues, that would effectively create these crucial faculties without actually needing to make deeper algorithmic progress. Like human computers in 17th century, these AIs might end up doing manually what a better algorithm could do at a much lower level, much more efficiently. But it would still be much more effective than when it doesn’t happen at all, and AI labor scales well.

Vladimir_Nesov 9 Dec 2025 22:25 UTC
2 points
0
in reply to: artifex0’s comment on: Little Echo

to rationally pre-commit to acting irrationally in the future

Like conflation around “belief”, it’s better to have a particular meaning in mind when calling something “rational”, such as methods that help more with finding truth, or with making effective plans.

(If there’s something you should precommit to, it’s not centraly “irrational” to do that thing. Or if it is indeed centrally “irrational” to do it, maybe you shouldn’t precommit to it. In this case, it’s only “irrational” according to a myopic problem statement that is itself not the right thing to follow. And in the above narrow sense of “rationality” as preference towards better methods rather than merely correctness of individual beliefs and decisions according to given methods, none of these things are either “rational” or “irrational”.)

Vladimir_Nesov 9 Dec 2025 3:11 UTC
11 points
5
on: Every point of intervention

Leaving aside tractability … neglectedness … and goodness … I wanted to argue for importance.

Neglectedness is an unusually legible metric and can massively increase marginal impact. So acute awareness of neglectedness when considering allocation of effort should solve most issues of failing to address every possible point of intervention. Assessment of tractability/goodness/importance gives different puzzles for every hypothetical intervention, and studying these puzzles can itself be a project. Neglectedness is more straightforward, it’s a lower-hanging strategic fruit, a reason not to skip assessing tractability/goodness/importance for things nobody is working on, to not dismiss them out of hand for that reason alone.

Vladimir_Nesov 8 Dec 2025 23:35 UTC
3 points
0
in reply to: β-redex’s comment on: Vladimir_Nesov’s Shortform
That things other than chips need to be redesigned wouldn’t argue either way, because in that hypothetical everything could just come together at once, the other things the same way as the chips themselves. The issue is capacity of factories and labor for all the stuff and integration and construction. You can’t produce everything all at once, instead you need to produce each kind of thing that goes into the finished datacenters over the course of at least months, maybe as long as 2 years for sufficiently similar variants of a system that can share many steps of the process (as with H100/H200/B200 previously, and now GB200/GB300 NVL72).

How elaborate the production process needs to be also doesn’t matter, it just shifts the arrival of the finished systems in time (even if substantially), with the first systems still getting ready earlier than the bulk of them. And so the first 20% of everything (at a given stage of production) will be ready partway into the volume production period (in a broad sense that also includes construction of datacenter buildings or burn-in of racks), significantly earlier than most of it.

Vladimir_Nesov 8 Dec 2025 20:25 UTC
3 points
0
in reply to: ryan_greenblatt’s comment on: Vladimir_Nesov’s Shortform
Once a model is trained, it needs to be served. If both xAI and OpenAI have their ~6T total param models already trained in Jan 2026, xAI will have enough NVL72 systems to serve the model to all its users at a reasonable speed and price (and so they will), while OpenAI won’t have that option at all (without restricting demand somehow, probably with rate limits or higher prices).

A meaningful fraction of the buildout of a given system is usually online several months to a year before the bulk of it, that’s when the first news about cloud access appear. If new inference hardware makes more total params practical than previous hardware, and scaling of hardware amount (between hardware generations) is still underway, then even a fraction of the new hardware buildout will be comparable to the bulk of the old hardware (which is busy anyway) in FLOPs and training steps that RL can get out of it, adjusting for better efficiency. And slow rollouts for RL (on old hardware) increase batch sizes and decrease the total number of training steps that fit into a few months, this could also be important.

serving to users might not be that important of a dimension

What new hardware becomes available before the bulk of it is not uniquely useful for serving to users, because there isn’t enough of it to serve a flagship model with more total params to all users. So it does seem to make sense to use it for RL training of a new model that will use the bulk of the same hardware for serving the model to users once enough of it is built.

Vladimir_Nesov 8 Dec 2025 0:15 UTC
6 points
0
in reply to: anaguma’s comment on: Vladimir_Nesov’s Shortform
I don’t think there is a delay specific to NVL72, it just takes this long normally, and with all the external customers Nvidia needs to announce things a bit earlier than, say, Google. This is why I expect Rubin Ultra NVL576 (the next check on TPU dominance after 2026′s NVL72) to also take similarly long. It’s announced for 2027, but 2028 will probably only see completion of a fraction of the eventual buildout, and only in 2029 will the bulk of the buildout be completed (though maybe late 2028 will be made possible for NVL576 specifically, given the urgency and time to prepare). This would enable companies like OpenAI (without access to TPUs at gigawatt scale) to serve flagship models at the next level of scale (what 2026 pretraining compute asks for) for all its users, catching up to where Google and Anthropic were in 2026-2027 thanks to Ironwood. Unless Google decides to give yet another of its competitors this crucial resource and allows OpenAI to build gigawatts of TPUs earlier than 2028-2029.

Vladimir_Nesov 7 Dec 2025 23:48 UTC
71 points
8
on: AI in 2025: gestalt
Pretraining (GPT-4.5, Grok 4, but also counterfactual large runs which weren’t done) disappointed people this year. It’s probably not because it wouldn’t work; it was just ~30 times more efficient to do post-training instead, on the margin. This should change, yet again, soon, if RL scales even worse.

Model sizes are currently constrained by availability of inference hardware, with multiple trillions of total params having become practical only in late 2025, and only for GDM and Anthropic (OpenAI will need to wait for sufficient GB200/GB300 NVL72 buildout until later in 2026). Using more total params makes even output tokens only slightly more expensive if the inference system has enough HBM per scale-up world, but MoE models get smarter if you allow more total params. At 100K H100s of pretraining compute (2024 training systems), about 1T active params is compute optimal ^[[1]] , and at 600K Ironwood TPUs of pretraining compute (2026 systems), that’s 4T active params. With even 1:8 sparsity, models of 2025 should naturally try to get to 8T total params, and models of 2027 to 30T params, if inference hardware would allow that.

Without inference systems with sufficient HBM per scale-up world, models can’t be efficiently trained with RL either, thus lack of availability of such hardware also results in large models not getting trained with RL. And since 2025 is the first year RLVR was seriously applied to production LLMs, the process started with the smaller LLMs that allow faster iteration and got through the orders of magnitude quickly.

GPT-4.5, Grok 4

GPT-4.5 was probably a compute optimal pretrain, so plausibly a ~1T active params, ~8T total params model ^[[2]] , targeting NVL72 systems for inference and RL training that were not yet available when it was released (in this preliminary form). So it couldn’t be seriously trained with RL, and could only be served on older Nvidia 8-chip servers, slowly and expensively. A variant of it with a lot of RL training will likely soon get released to answer the challenge of Gemini 3 Pro and Opus 4.5 (either based on that exact pretrain, or after another run adjusted with lessons learned from the first one, if the first attempt that became GPT-4.5 was botched in some way, as the rumor has it). Though there’s still not enough NVL72s to serve it as a flagship model, demand would need to be constrained by prices or rate limits for now.

Grok 4 was the RLVR run, probably over Grok 3′s pretrain, and has 3T total params, likely with fewer active params than would be compute optimal for pretraining on 100K H100s. But the number of total params is still significant, so since xAI didn’t yet have NVL72 systems (for long enough), its RL training wasn’t very efficient.

This should change, yet again, soon

High end late 2025 inference hardware (Trillium TPUs, Trainium 2 Ultra) is almost sufficient for models that 2024 compute enables to pretrain, and plausibly Gemini 3 Pro and Opus 4.5 already cleared this bar, with RL training applied efficiently (using hardware with sufficient HBM per scale-up world) at pretraining scale. Soon GB200/GB300 NVL72 will be more than sufficient for such models, when there’s enough of them built in 2026. But the next step requires Ironwood, even Rubin NVL72 systems will constrain models pretrained with 2026 compute (that want at least ~30T total params). So unless Google starts building even more giant TPU datacenters for its competitors (which it surprisingly did for Anthropic), there will be another period of difficulty with practicality of scaling pretraining, until Nvidia’s Rubin Ultra NVL576 are built in sufficient numbers sometime in late 2028 to 2029.
1. ↩︎
  Assuming 120 tokens/param compute optimal for a MoE model at 1:8 sparsity, 4 months of training at 40% utilization in FP8 (which currently seems plausibly mainstream, even NVFP4 no longer seems completely impossible in pretraining).
2. ↩︎
  Since Grok 5 will be a 6T total param model, intended to compete with OpenAI and targeting the same NVL72 system, maybe GPT-4.5 is just 6T total params as well, since if GPT-4.5 was larger, xAI might’ve been able to find that out and match its shape when planning Grok 5.
What links here?

Vladimir_Nesov 7 Dec 2025 5:42 UTC
97 points
0
on: Vladimir_Nesov’s Shortform
Flagship models need inference compute at gigawatt scale with a lot of HBM per scale-up world. Nvidia’s systems are currently a year behind for serving models with trillions of total params, and will remain behind until 2028-2029 for serving models with tens of trillions of total params. Thus if OpenAI fails to access TPUs or some other alternative to Nvidia (at gigawatt scale), it will continue being unable to serve a model with a competitive amount of total params as a flagship model until late 2028 to 2029. There will be a window in 2026 when OpenAI catches up, but then it’s behind again.

The current largest flagship models are Gemini 3 Pro and Opus 4.5, probably at multiple trillions of total params, requiring systems with multiple TB of HBM per scale-up world to serve efficiently. They are likely using Trillium (TPUv6e, 8 TB per scale-up world) and Trainium 2 Ultra (6 TB per scale-up world), and need north of high hundreds of megawatts of such systems to serve their user bases.

Nvidia’s system in this class is GB200/GB300 NVL72 (14/20 TB per scale-up world), but so far there isn’t enough of it built, and so models served with Nvidia’s older hardware (H100/H200/B200, 0.6-1.4 TB per 8-chip scale-up world) either have to remain smaller or become more expensive. The smaller amount of NVL72s that are currently in operation can only serve large models to a smaller user base. As a result, OpenAI will probably have to keep the smaller GPT-5 as the flagship model until they and Azure build enough NVL72s, which will happen somewhere in mid to late 2026 (the bigger model will very likely get released much earlier than that, perhaps even imminently, but will have to remain heavily restricted by either price or rate limits). Paradoxically, xAI might be in a better position as a result of having fewer users, and so they might be able to serve their 6T total param Grok 5 starting early 2026 at a reasonable price.

But then in 2026, there is a gigawatt scale buildout of Ironwood (TPUv7, 50 TB per scale-up world for the smallest 256-chip pods, more with slices of 9216-chip pods), available to both GDM and Anthropic. This suggests the possibility of flagship models (Gemini 4, Opus 5) with tens of trillions of total params at the end of 2026 (maybe start of 2027), 10x larger than what we have today, something even Nvidia’s NVL72 systems (either Blackwell or Rubin) won’t be as suited to cope with. Nvidia’s answer to Ironwood is Rubin Ultra NVL576 (150 TB of HBM per scale-up world), but it’s only out in 2027, which means there won’t be enough of it built until at least late 2028, plausibly 2029 (compare to GB200 NVL72 being out in 2024, with gigawatt scale systems only built in 2026). So if OpenAI/xAI/Meta want to serve a flagship model with tens of trillions of total params in 2026-2028, they need access to enough TPUs, or some other alternative systems (which seems less likely given the short notice).
What links here?
- Vladimir_Nesov's comment on AI in 2025: gestalt by technicalities (7 Dec 2025 23:48 UTC; 71 points)