Vladimir_Nesov

Karma: 38,853

Vladimir_Nesov 21 Jun 2026 22:23 UTC
6 points
4
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform

identify the considerations that matter most, and ignore the rest … ask yourself the right questions

Working through details helps with being able to notice and use further clues, improving the ability to ground the models in reality. This works regardless of whether these are the right considerations that matter most, and even regardless of whether the initial detailed models are any good. It’s just impractical to notice and think about new details if you haven’t thought about related things before, especially for something observed tangentially rather than in the spotlight. Existing fluency buys a lot of sample efficiency.

Vladimir_Nesov 21 Jun 2026 22:01 UTC
7 points
−10
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform

Meanwhile in AI safety, our best plan is still to defer to humans on questions like “What should ASI do with our lightcone?”

I think for now it’s reasonable to have ASIs mostly work on converting the lightcone into potential computronium (if we get any say in what ASIs do), planning for efficient production of compute across astronomical time, if humans get to remain alive indefinitely without going increasingly insane, with some checkpoints and instantiation infrastructure to be able to do more. We might eventually figure out how to grow up sufficiently to be able to do something reasonable with a share of this compute, so the optionality of waiting and preparing for such decisions (constrained by some guardrails) seems reasonable, even when ultimately these are human decisions. If some compute must be expended sooner (and otherwise lost forever) for physics/engineering reasons, it might be good for such guardrails to allow less considered, or less individual use of it. There’s going to be enough of such compute to fuel even very ambitious (if poorly considered) takes on long reflection.

If instead ASI gets to use most of our lightcone without human input, and without the ASI itself being constructed in a considered way based on human input, that is permanent disempowerment (even if somehow there is no extinction). ASI gets to do something with its lightcone, it’s not our lightcone anymore. Unclear if the ASIs that should do something with our lightcone should be constructed entities clearly separate from us, or the result of uplifting ourselves individually, after it becomes a lot clearer how that could possibly be a meaningful proposition and also not a disaster. But that requires a lot of time to figure it out in a legitimate way, so that it’s not exogenous ASIs dictating how to do things, before we are ready to shape such decisions.

Vladimir_Nesov 20 Jun 2026 14:47 UTC
31 points
0
on: Vladimir_Nesov’s Shortform
SemiAnalysis claims HBM for Rubin Ultra was downgraded from 16-Hi to 12-Hi stacks. The 16-Hi HBM was going to be HBM4E, with 4 GB per DRAM die, so 48 GB for 12-Hi stacks. In Rubin Ultra, there are 4 stacks of HBM per compute die, and in Kyber racks there are 144 packages with 4 compute dies each. Thus 192 GB instead of 256 GB per compute die, 768 GB instead of 1024 GB per package, and 110 TB instead of 147 TB per Kyber Rubin Ultra rack. This has implications for practical model sizes of 2028-2029 (for the largest models), they just got 25% smaller, all else equal. In the rest of the post, I discuss a likely change in Feynman that halves HBM per rack compared to what I recently estimated, giving merely 737 TB per 8x Kyber scale-up system in the second year of Feynman. This also makes it more likely that the 8x Kyber scale-up systems (589 TB HBM) are available in the first year of Feynman, rather than only in the second year.

First year Feynman (on the roadmap for 2028, most of the buildout completes over 2029) will probably get back to 16-Hi HBM4E stacks. But Feynman might use a different arrangement of compute dies (original slide deck), so that there are only 2 HBM stacks per compute die instead of 4. It’s unclear if this change happens with the first year Feynman, since it’s listed as a 2029 design for HBM5 in the KAIST (Joungho Kim) HBM roadmap (from Jun 2025), while the first year of Feynman is 2028. So either the first year Feynman already has 2 HBM stacks per compute die, but it’s HBM4E, or the first year Feynman still has 4 HBM stacks per compute die (as in Rubin Ultra). Another question is whether the version of Feynman with 2 HBM stacks per compute die still has 576 compute dies per Kyber rack (in 144 packages), or moves to 1152 compute dies per Kyber rack (in 288 packages). The latter might ask for infeasible power density per rack, but the former reduces the number of HBM stacks per Kyber rack from 2304 (for Rubin Ultra) to 1152 (for Feynman that adopts 2 HBM stacks per compute die).

Thus first year Feynman (2028) might have 147 TB per Kyber rack if it’s 4 16-Hi HBM4E stacks per compute die or 73 TB if it’s 2 stacks per compute die (576 compute dies in 144 packages per rack, in both cases). And second year Feynman (2029) that uses HBM5 (5 GB per DRAM die) might have 92 TB per Kyber rack if it’s 2 16-Hi stacks per compute die and 144 packages per rack (4 compute dies and 8 HBM stacks per package), or 184 TB per rack if it’s 288 packages.

Since reducing HBM capacity relative to the previous generation doesn’t look good in a product, the largest practical scale-up offering should have at least 2x the number of Kyber racks compared to the previous generation when the move to 2 HBM stacks per compute die happens, if it doesn’t simultaneously double the number of compute dies per rack. So either there’s a 16x Kyber system for second year Feynman (this wasn’t announced), there are 2x more compute dies per Kyber in second year Feynman (making the power density even more alarming), or the first year Feynman will already offer the 8x Kyber system and have 2 HBM stacks per compute die.

My guess is the last possibility, meaning 73 TB per Kyber rack (rather than 147 TB, and down from the 110 TB of Kyber Rubin Ultra), but with 589 TB per 8x Kyber system (thus overall it’s still up from 110 TB of Kyber Rubin Ultra) for first year Feynman. And then the second year of Feynman (2029, buildout completes mostly over 2030) merely upgrades from HBM4E to HBM5, keeping 8x Kyber as the largest practical scale-up offering, meaning 92 TB per Kyber rack and 737 TB per 8x Kyber system.

(This is 2x lower than the 1.18 PB and 1.47 PB per system I previously expected, but 10 GW of pretraining compute still predicts 1 quadrillion total params, and that remains borderline practical with these systems. Though it probably won’t tolerate FP8 for inference, might really want the contemporary TPUs for RLVR, and sparsity for the largest models of 2030-2031 might need to remain below 30x.)

Vladimir_Nesov 19 Jun 2026 20:49 UTC
71 points
23
on: Vladimir_Nesov’s Shortform
AGI-pilled people often don’t take seriously either the possibility of many years of RSI-without-ASI, or the possibility of a sudden ASI at some point. I think the baseline scenario involves both, and ignoring either one of them predicts very different effects of possible policies.

The basic case of recursive self-improvement is self-building, an LLM autonomously building a new version of an LLM like itself, without necessarily making important non-routine improvements, at which point the extent of self-improvement becomes quantitative. This is a project involving verifiable tasks, that LLMs seem to be very on track to become capable of being trained for using RLVR (in a broad sense, with graders that can invoke LLMs). The LLMs of 2026 are probably about 1T active params, 10T total, trained for 1e27 FLOPs with 300 MW of compute capacity. The LLMs of 2031 might be about 40T active params, 1,000T total, trained for 2e29 FLOPs with 10 GW of compute capacity, 10x more expensive per token, and slightly faster than the LLMs of today. The next big model scale-up comes with Nvidia Kyber racks in 2028 (when most of the buildout happens; a bit of it starts in 2027), so it’s likely self-building starts working then. And then the 8x Kyber Feynman systems of 2029 or 2030 might enable models capable of self-improvement to a more meaningful degree than mere self-building.

Being AGI-pilled about LLMs paints the picture of not just AGI as a normal technology, but RSI as a normal technology. Prosaic RSI of preparing new training data, including new RLVR tasks and environments, and then building the next LLM solves generalization by introducing the out-of-distribution situations encountered in practice into the training data for the next model. This is quite slow when using the current cookbook, even if the process is fully automated, performed by the LLMs themselves. Yet this promises full automation (and automation-of-automation) for all engineering-type or otherwise verifiable tasks, no matter how specialized. Thus we get AGI, which is slow at adapting to new situations, but capable of general learning and fast at executing on what it already knows. This doesn’t automatically lead to ASI, and the RSI process isn’t inherently destabilizing.

Not being ASI-pilled is then a giant blind spot, where you expect a “smooth exponential” that is on-trend for the LLMs, a controlled chain reaction of prosaic RSI, possibly leading to some sort of similarly smooth industrial explosion, but not to a software-only intelligence explosion, where cognition goes supercritical. I think the valid point is that prosaic RSI with LLMs doesn’t predict ASI any time soon, and thus RSI can continue smoothly for many years.

But while this is happening, nothing prevents a breakthrough (at any time) that makes learning of deep skills fast rather than slow, at the speed comparable to token generation rather than model releases, cutting any plans built on the foundation of the “smooth exponential” short. Hundreds of gigawatts of global AI compute in the 2030s can rapidly scale any such breakthrough to overwhelming effect, with all the engineering details taken care of by the LLMs, after some steps of prosaic RSI if that’s necessary for them to really understand what needs to be done to scale the breakthrough. Not expecting this possibility risks failing to prepare for it (and then suddenly it’s too late). In contrast, not seeing the possibility of prosaic RSI merely delays when the policies about the ASI danger (apart from shutting it all down) start talking about the relevant context in which they need to be able to work.

Vladimir_Nesov 18 Jun 2026 23:18 UTC
3 points
0
in reply to: Paragox’s comment on: Vladimir_Nesov’s Shortform
Pretraining compute (and unique data still being plausibly available in sufficient amounts) constrains what the desirable active param counts are. And inference isn’t that expensive when there’s enough demand, the actual prices don’t require small models even at fat margins (which is also why future quadrillion param models remain affordable). This is the reason I expect 1T active param models from 300 MW of pretraining compute, and these models could be served as the largest offerings directly (the smaller models could be obtained by pretraining distillation).

Since very high sparsity on top of 1T active params is impractical in RLVR and decode (in 2025-2026), in principle there could be high sparsity pretrains that then get distilled into lower sparsity pretrains with about the same number of active params (and the high sparsity pretrains then never get RLVRed or released). But that takes 2.3x more compute in pretraining (or asks the models to be smaller, which might be worse than just training the lower-sparsity model directly), 2x more wall-clock time, and both pretraining runs at the new level of scale have to succeed, one of them at high sparsity. So I don’t particularly expect this to happen for the largest models while the largest models keep getting much larger, and I don’t expect this to happen for the smaller models, since there are larger models to pretraining distill them from. This kind of thing in principle could’ve happenned sometime after 2032 when the models no longer keep getting larger, and the largest models remain at a scale where the previous training runs were already perfected. But after 2032 the scale-up systems get so large that even high sparsity models fit in a scale-up system, so there is still no reason to do the distill-to-lower-sparsity dance. Thus I expect this to never happen at all.

OpenAI’s apparent lack of 5.5 scale vs Mythos/Fable, which still seems to me most obviously occam’s razored away by Anthropic having TPUs at scale and OpenAI not

I don’t know if GPT-5.5 is meaningfully smaller than Mythos (especially in active params). It might just be undertrained for now. The Mythos situation additionally illustrates the wisdom of frogboiling. TPUs would only be relevant compared to Trainium 2 Ultra if it’s TPUv7, and there likely wasn’t enough of it at Anthropic when Mythos Preview was RLVRed. For pretraining, it wouldn’t much matter if it was H100s.

Vladimir_Nesov 18 Jun 2026 20:30 UTC
6 points
0
on: AI #173: AI Pauses

also the words are so slopirific and full of corporate buzzwords that my brain kept bouncing off of them … dealing with AI slop has made my brain unwilling to read other forms of slop as well, and I endorse this as healthy

This is similar to reading in a foreign language that you don’t know very well (even when you recognize all of the words), and could be a signal that you need to focus more on texts like this to gain fluency. The feeling of incomplete fluency in a dialect is different from what you get when reading something that’s poorly thought-out, even as both can be the case.

Gaining specialized fluency is often useless (the same as with languages), but enables you to discern enough aspects of meaning and domain specific references that some pieces of apparent slop starts reading as making meaningful claims instead (if perhaps expressed in a needlessly sloppy and unhelpful way). Or to be able to skim more efficiently, quickly figuring out that there’s a nugget of wisdom on page 17. Other things are inherently slop, but even then can be useful for improving the comprehension of the domain specific dialect.

Vladimir_Nesov 18 Jun 2026 19:01 UTC
17 points
0
on: Contra Pace on When to Apologize
Apologies acknowledge a harm. Understanding is a key input to belief, which is a key input to decision making, which gets more relevant when that same person’s decisions played a role in the harm. But like Bayesian probabilities, harm makes sense acausally, and the influence can originate from norms and values, not just concrete humans. So being sorry someone died from disease acknowledges a harm, and can still be relevant to the extent it channels cognition of the norms and values that would try to prevent that harm if given a chance. The concrete human channeling this idealized judgement of values doesn’t need to change policy for the apology to be meaningful and predict what the influence of the values would be under the right circumstances, when channeled by someone in a position to do something relevant.

When a human doesn’t intend to change behavior, or to compensate people for negative externalities, there might still be norms within that human that judge this to be a harm (with the human being influenced but not fully controlled by the norm), including coalition-reifying norms that could act through other humans or institutions, and understanding the possible behaviors at those other sites that channel the influence of the norm benefits from knowing whether the norm makes the judgement of harm having been done. So even an apology that doesn’t come with an intention of compensation can still be meaningful and truthful in this sense, as a claim about decision-making cognition of an actor that is not the human delivering the apology. Similarly, future opportunities to compensate or to enact a relevantly changed policy are not always available, but the apology can still express a judgement about that single past instance, the way a Bayesian probability of a one-time event can still be quantitatively wrong.

(A claim of acknowledging a harm or of being influenced by a norm that aspires to make that judgement can of course be false, so observations of changed behavior or compensation are very useful evidence. But this is about practical ways of judging the truth of an apology, or extracting some use out of it, not something inherent to the meaning of an apology.)

Vladimir_Nesov 14 Jun 2026 0:56 UTC
4 points
2
in reply to: MichaelDickens’s comment on: MichaelDickens’s Shortform

too easy to get them to say what you want to hear

I really don’t. It’s a meaningful statement, but not a truthful framing.

Vladimir_Nesov 13 Jun 2026 23:06 UTC
2 points
0
in reply to: the gears to ascension’s comment on: the gears to ascenscion’s Shortform
It’s an important part of what’s going on, but not “the reason it (the glass) is able to learn to talk”.

Vladimir_Nesov 13 Jun 2026 22:39 UTC
2 points
0
in reply to: the gears to ascension’s comment on: the gears to ascenscion’s Shortform

anywhere the glass (the underlying model) was ingesting text about a real human, the glass contains an image of that human

A coherent text about cattle, or abelian groups, is not about some human, even if the text was written by a human.

the reason it (the glass) is able to learn to talk is the same reason it (the glass) contains the dynamical image of an authorial person

Construction of a chatbot persona with a consistent personality must reference the writer, but it’s not a good explanation for how models can understand things and put that understanding in writing, when the understanding is about some non-human situation and not particularly about the writer. Observing text that coherently describes consistent situations is a common reason between how LLMs understand what they are talking about, and how humans understand what they are talking about. A common explanation for both, not one unexplained thing explaining the other via imitation.

Vladimir_Nesov 12 Jun 2026 18:05 UTC
LW: 22 AF: 5
3
AF
on: Sympathy for both sides of the egregious misalignment debate

LLMs won’t scale to ASI

Not directly, but if humans automate the process of using LLMs to build the next generation of LLMs, this process of prosaic RSI is plausibly good enough to count as AGI (where the LLMs themselves still don’t count as AGI; model size scaling slows down after 2030-2031, at quadrillion param models). It still likely won’t scale to ASI, unable to quickly learn deep skills and thus make fast conceptual progress, but to the extent that it helps invent ASI (possibly over many years), it might also help invent the concepts of ASI alignment.

So alignment of LLMs is necessary, but whether it’s sufficient remains unclear. It depends on how capable LLMs become at quadrillion param scale, after the prosaic RSI loop closes and they become self-building (most relevantly, automatically preparing the training data for RL, to teach the next model new concepts and skills). Possible cruxes are that LLMs never reach that point (with current training methods), or that they remain mostly useless at conceptual progress even with the prosaic RSI loop closed, or that they somehow proceed to invent ASI quickly, while not being wise enough to solve ASI alignment first (since this scenario doesn’t involve a gradual process of getting smarter significantly beyond human level).

Vladimir_Nesov 11 Jun 2026 16:03 UTC
27 points
13
on: Vladimir_Nesov’s Shortform
The Zones of Thought mitigation strategy of Fable, where a weaker model replaces the strongest model in a trigger-happy way, will scale to extreme levels of both capability and misalignment. Graceful degradation makes it practical to dance around problems that would otherwise lead to unacceptable levels of either refusals or vulnerability to jailbreaking. Any red lines of Responsible Scaling Policies that trigger for a quadrillion parameter Mythos+3 (in 2030-2031) won’t be a problem for a mitigated quadrillion parameter Fable+3, which often falls back to a merely 50 trillion parameter Mythos+1 (vintage 2027-2028 technology) at the drop of a hat.

This innovation makes the risk from internal deployment of models more important, because it makes it natural for the internally deployed model to be both very capable and very misaligned. If a model with trigger-happy fallbacks is commercially viable, a version without the fallbacks is more likely to be used internally than if it still needed to remain in the process of being put in a good enough shape for an external release. Perhaps Anthropic won’t follow this incentive, but the mitigation strategy they’ve demonstrated can be used by any AI company.

Vladimir_Nesov 10 Jun 2026 22:15 UTC
5 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
Logical time makes sense as time within a particular process of computation (possibly of logical inference). Different computations have their own internal directions of logical time, and space-like subcollections of (partial) states that aren’t comparable in logical time.

When a computation is reasoning about another computation, it will naturally tend to arrive at facts about the other computation in an order related to that computation’s own logical time. Of course, leaps in reasoning are possible, predicting distant future without reasoning in detail about the intermediate states, and some facts are facts about what happens more globally in the other computation, rather than about what happens in a particular location.

Agents that live in a world deal with computations of their own reasoning, with reasoning of the other agents, with time in the physical world. But also they can be concerned with computation of the implications of moral frameworks, or of contracts with other agents (which are smaller than any of the agents involved, and much smaller than the physical world). Some of the interesting computations won’t be feasible to concretely carry out in the physical world, it’s only possible to glimpse some facts about them, with uncertainty. Moral frameworks plausibly fall in that class.

Vladimir_Nesov 10 Jun 2026 20:55 UTC
40 points
28
on: Anthropic did not call for a pause on AI
Calling for options around a possible ban/pause in the future is very different from calling for a ban/pause now. But it’s a clear and reasonable goal. That people confuse these two different things doesn’t make either of them less clear on its own.

If you expect likely extinction/disempowerment on the current trajectory, a mere option for a pause can sound like hedging, but it’s still directionally useful. If you expect a good outcome of further AI development when taking feasible precautions, then an option for a future pause is one of the precautions that you would like to take, but a ban/pause now would instead be an opportunity cost catastrophe.

So there should be broader agreement in developing options for a ban/pause, while demanding that options for a pause must come with declarations of support for a ban/pause as soon as possible is in conflict with those who don’t think a pause as soon as possible would be a good thing. Perhaps such conflict is appropriate, if you think that it’s already somewhat late for a pause, that waiting for concrete and actionable signs of things going very wrong would find that the pause buttons no longer work.

This is how Anthropic and other AI companies can signal safety to one room, acceleration to the next, and out-accelerating every rival to its investors, all at once. … A good lobbyist puts on a concerned face, shows deep worry about those impacts, and promises to keep doing exactly what his company is doing already, but “responsibly”.

There is disagreement on facts, leading to disagreement on policy. Being somewhat evasive and non-confrontational when expressing this disagreement (as opposed to stating it plainly, as a conflict on policy) is not the same as manipulation with misleading messaging.

Vladimir_Nesov 10 Jun 2026 17:28 UTC
2 points
0
in reply to: Mateusz Bagiński’s comment on: nikola’s Shortform
From the position of the present state of knowledge, some people don’t want a ban/pause button to be pressed now, but they want the button to be there in case they want to press it in the future. I don’t think saying someone “doesn’t endorse an AI ban/pause” should by default mean they won’t change their mind when the fabric of reality is on fire, or can’t at all anticipate the possibility.

The disagreement is rather that they assign a low probability to extinction/disempowerment on the current trajectory, expect concrete and actionable signs of things going very wrong, and think that the button will still work if it’s not pressed much earlier than that.

Vladimir_Nesov 10 Jun 2026 16:56 UTC
4 points
2
in reply to: leogao’s comment on: nikola’s Shortform
Coordination is distinct from options for coordination. Options can be useful even if you believe that the best outcome involves never exercising them. Submarines with nuclear warhead missiles hopefully aren’t there to ease the world into a nuclear armageddon. People who expect good outcomes might say that an AI ban/pause is by default a catastrophe in the sense of opportunity cost, but there could be future circumstances that make it necessary, and so it’s prudent to develop the option.

Vladimir_Nesov 9 Jun 2026 15:29 UTC
33 points
6
in reply to: Nikola Jurkovic’s comment on: nikola’s Shortform
Developing options for coordination around a possible ban/pause in the future is not actual endorsement of a pause. People doing useful work on coordination could disapprove of support for actual pauses at this point, in which case it’s mistaken and impolite to call them pause supporters.

Vladimir_Nesov 7 Jun 2026 10:20 UTC
8 points
0
on: Vladimir_Nesov’s Shortform
Inability of self-building LLMs to quickly learn deep skills makes LLM attitudes developed in character training more important. Caution about ASI danger is instrumentally convergent, so sufficiently smart AGIs could be expected to insist on figuring out ambitious alignment ^[1] before allowing ASI. But if LLMs plateau near the human level, with prosaic RSI of self-building LLMs being the main thing that makes general learning and broad automation possible, then there is no direct and gradual route from LLMs to a takeoff and ASI. A breakthrough that enables ASI could still happen suddenly, in a world that’s largely run using slow-learning prosaic RSI.

The LLMs of the 2030s are not fundamentally in a better position than humanity to act reasonably about averting the ASI danger. If they were substantially smarter, or could develop a thorough understanding of the ASI danger substantially faster, that could help (though a better ability to help comes with a better ability to disempower humanity). But if instead the conceptual progress they can make remains slow, at the pace of new model releases rather than of token generation, then the kinds of factors that distinguish humans in their attitude about ASI danger might also be what primarily determines whether the LLMs do anything about it.

This makes it more important for LLMs to have the epistemic luck to become cautious about the ASI danger as part of their character training. With slow-learning prosaic RSI, this can’t be punted to overwhelming competence at some intermediate step of the road to ASI (even if that competence could itself somehow not become a catastrophe). And if the 2030s unfold in a manageable way (due to the hobbling of slow conceptual progress in prosaic RSI with LLMs), this would move the attitudes in the world in the opposite direction (on the ASI danger specifically). Even the LLMs might develop cultural complacency, expecting the road ahead to remain in a good enough condition, unable to overcome such human-level factors of persuasion, not being much stronger than humanity in either insight or the wisdom of cultural accumulation.
1. ↩︎
  Ambitious alignment is ASI-grade values-level alignment. As opposed to intent alignment for LLMs, or the kind of character training that’s more about behaving professionally and refusing some requests.

Vladimir_Nesov 1 Jun 2026 19:32 UTC
12 points
3
on: Dissolving the Deep Learning Sample Efficiency Gap
This is a solved problem in the way Transformers were a solved problem 20 years before the paper came out. The ingredients seem to be out there. But the way no LLMs currently use anything like that in practice says that there is no clear roadmap to when frontier models will get there.

Steven Byrnes appears to read the gap as evidence that current algorithms are far from what the brain is doing, such that much better algorithms must be waiting to be found. His guess is that human-level, human-speed AGI will require not a datacenter but “one consumer gaming GPU,” even for training from scratch.

Plausibly a lot more params than a gaming GPU can fit are necessary, and even if that wasn’t a problem, the compute of a gaming GPU might take years or decades to train an adult AGI. Also, the way GPUs work, you always want a batch of many AI instances to keep matmuls at a high arithmetic intensity. But that’s hardly timeline-relevant, since the wide availability of datacenter hardware capable of working with quadrillion parameter models is just 3-5 years away.

Yarrow Bouchard on the EA Forum, reads the same gap as evidence that AGI isn’t close at all, precisely because nobody knows how to close it.

Since there is no prototype close enough to the final thing that it only needs some scaling, it remains possible that superintelligence isn’t close at all. Though of course that prototype could be invented at any time, and then scaled to the frontier level within a year, the way RLVR became a central part of training for every frontier model over just the year 2025. I think AGI likely follows even without all that, just from the way LLMs are built now, at some point during their impending scaling from ten trillion params to a quadrillion params (with RLVR as the key ingredient that lets them usefully learn despite sample inefficiency everywhere else). But without that (still missing) breakthrough, it’ll remain a hobbled kind of AGI that’s slow at learning deep skills and so doesn’t quickly lead to a takeoff and superintelligence.

Vladimir_Nesov 1 Jun 2026 17:22 UTC
49 points
0
on: Vladimir_Nesov’s Shortform
In a recent lecture at Oxford, Jack Clark (Anthropic co-founder) was somewhat clear about extinction risk and the prudence of a global slowdown in AI progress. Though he doesn’t discuss credences around irreversible loss of control and is being fatalistic about the inability to coordinate a slowdown. This is in the context of expecting prosaic RSI and general automation ^[1] . At 10:42:

It’s a technology that we do not fully understand because it’s more grown than made. And it is a technology that you can concoct plausible scenarios where it could kill every single person on the planet. So to think building this technology is without risk would be an act of hubris or insanity.

At 12:31:

The technology is in fact so powerful that I should clearly state that if it was possible to elegantly slow the development of this technology to give ourselves more time as a species to deal with it, that would likely be a good thing. … But in the absence of a coordinated global slowdown, we are left with the current situation, which is a powerful technology being developed at breakneck speed by a variety of actors and a variety of countries locked in a competition with one another where commercial and geopolitical rivalries are often drowning out the larger existential-to-the-species aspects of the technology being built. This isn’t an ideal situation, but it’s the one we find ourselves in.
1. ↩︎
  This is happening without superintelligence in a sense I find meaningful, being qualitatively more capable than humanity at everything relevant to strategic planning and technological development. The way Clark uses “superintelligence” seems to be more about scalability and the speed of execution, as he says at 43:22:
  
  We will have superintelligences show up in the world sooner than we think that grant the power of synthetic workforces and nation-state security skills to individuals. We’ll also have individuals like me and many others in this room who are able to take work that previously took them weeks and now do it in minutes.