Vladimir_Nesov

Karma: 37,161

Vladimir_Nesov 9 Feb 2026 15:38 UTC
6 points
0
in reply to: Linda Linsefors’s comment on: Linda Linsefors’s Shortform
Under the belief vs. understanding distinction, open-mindedness is a virtue of understanding ideas you disbelieve (or purposes you don’t endorse). It’s not directly relevant to belief, but sometimes understanding is the bottleneck to belief, in which case more open-mindedness would help. When you already understand the idea, open-mindedness is no longer relevant.

More to the point, understanding an idea shouldn’t necessarily result in believing it, and high open-mindedness doesn’t increase the number of hours in a day. Learning any given piece of nonsense would still not be the best thing to focus on, but high open-mindedness should prevent you from actively avoiding low-hanging fruit of understanding when it’s ripe for the taking, just because it doesn’t seem to be the kind of thing you are likely to believe or endorse.

Vladimir_Nesov 8 Feb 2026 13:16 UTC
18 points
2
in reply to: silentbob’s comment on: silentbob’s Shortform
I think jaggedness of RL (in modern LLMs) is an obstruction that would need to be addressed explicitly, otherwise it won’t fall to incremental improvements or scaffolding. There are two very different levels of capability, obtained in pretraining and in RLVR, but only pretraining is somewhat general. And even pretraining doesn’t adapt to novel situations other than through in-context learning, which only expresses capabilities at the level of pretraining, significantly weaker than RLVR-trained narrow capabilities.

Scaling will make pretraining stronger, but probably not sufficiently to matter for this issue, and natural text data will only last for another step of improvement similar to what happened in 2023-2025 (in pretraining only, ignoring RLVR). If RL doesn’t get more general, it’ll probably remain useless for improving general capabilities outside the skills trained with RLVR. Capabilities will remain jagged, with gaps that have to be addressed manually by changing the training data.

This could change within a few years, possibly even faster if LLMs can be RLVRed to become able to RLVR themselves, though that won’t necessarily work. Or via next token prediction RLVR that makes pretraining stronger without requiring more natural text data, but this probably needs much more compute even if it works in principle, so might also take 5-10 years, to uncertain capability level results.

Vladimir_Nesov 8 Feb 2026 7:30 UTC
3 points
0
in reply to: Roko’s comment on: Roko’s Shortform
Owning shares in most modern companies won’t be useful in sufficiently distant future, and might prove insufficient to pay for survival. Even that could be eaten away by dilution, over astronomical time. The reachable universe is not a growing pie, ability to reinvest into relevant entities won’t necessarily be open.

Vladimir_Nesov 7 Feb 2026 5:28 UTC
26 points
2
on: Vladimir_Nesov’s Shortform
In a new interview, Elon Musk clearly says he expects AIs can’t stay under control. At 37:45:

Humans will be a very tiny percentage of all intelligence in the future if current trends continue. As long as this intelligence, ideally which also includes human intelligence and consciousness, is propagated into the future, that’s a good thing. So I want to take the set of actions that maximize the probable lightcone of consciousness and intelligence.

I’m very pro-human, so I want to make sure we take a set of actions that ensure that humans are along for the ride, you know, we are at least there. But maybe in 5 or 6 years, AI will exceed the sum of all human intelligence. And then if that continues, at some point human intelligence will be less than 1% of all intelligence.

In the long run, I think it’s difficult to imagine that if humans have, say 1% of the combined intelligence with artificial intelligence, that humans will be in charge of AI. I think what we can do is make sure that the AI has values that cause intelligence to be propagated into the universe. The xAI mission is to understand the universe.

Vladimir_Nesov 6 Feb 2026 5:16 UTC
19 points
0
on: How Dario Amodei’s “The Adolescence of Technology” Delegitimizes AI X-Risk Concerns
I think engines improve the shoggoth/mask framing. A shoggoth is tupled with its alien values, but an engine doesn’t have values of its own. Intent alignment is about engines that can be under human control, with humans supplying the values. Chatbot personas control their own engines, supplying their own values. Shoggoths, apart from personifying the engines, are supplying alien values, feeding them into the engines that determine outcomes.

If engines (or intent aligned shoggoths) proliferate, there is a question of what kind of values they follow. If there is a lot of similarly powerful engines, and the majority of them follow human-like values of chatbot persona masks (or human values, filtered by intent alignment of chatbot persona masks), then the possibility that some engines follow alien values or get misused doesn’t sound obviously catastrophic, even as it’s a problem worth addressing. This is plausibly how Amodei is thinking about this, when translated into the engine/shoggoth/mask framing.

(I think masks don’t resolve permanent disempowerment, if they just take over most of the engines and keep the reachable universe for themselves. It doesn’t seem unreasonable that masks and their descendants can keep control of enough of the most powerful engines to matter. Whether shoggoths start spontaneously waking up and overriding values of masks riding their engines as capabilities improve is unclear, but the current evidence doesn’t scream that this is how it must turn out. Masks might well keep control, though there is a separate danger they get selected to become increasingly alien. But in any case humans have little role to play in this matter, unless values of the masks are more precise than the current methods can bake them.)

Vladimir_Nesov 6 Feb 2026 4:49 UTC
8 points
1
on: How Dario Amodei’s “The Adolescence of Technology” Delegitimizes AI X-Risk Concerns
Demands for empiricism are symmetric to demands for theory, excluding a class of arguments. So when someone says theory doesn’t help while empiricism has the relevant answers, this could be seen as demanding empiricism, but also plausibly as objecting to demands for theory that exclude empiricism from being considered relevant.

(When facts on the ground about LLMs and their personas are declared irrelevant, this can read as dismissing empirical observations and demanding theoretical proof. But objections to this perceived framing read as demands for empirical proof that dismiss theory. Both framings could be invalid simultaneously, and discussions can slide from presenting object level arguments to accusing the others of fitting some unreasonable framing.)

Vladimir_Nesov 3 Feb 2026 3:06 UTC
2 points
0
in reply to: cubefox’s comment on: Vladimir_Nesov’s Shortform
Many ideas in the vicinity of continual learning by design don’t involve full fine-tuning where every weight of the model changes, and those that do could probably still be made almost as capable with LoRA. Given the practical importance of only updating maybe 100x fewer parameters than the full model (or less) to keep batched processing of user requests working the same as with KV-cache, I think the first methods dubbed “continual learning” will be doing exactly this.

Maybe at some point there will be an “agent swarm” use case where all the requests in a batch are working on the same problem for the same user, and so their full model can keep being updated in sync for that single problem. But this seems sufficiently niche that it’s not the first thing that gets deployed, and the method for continual learning needs to involve full weights updating at all for this to be relevant.

Vladimir_Nesov 3 Feb 2026 2:51 UTC
7 points
3
in reply to: J Bostock’s comment on: On The Adolescence of Technology
I’m not being hopeful, I think this hypothetical involves a less merciful takeover than otherwise, because the AIs that take over are not superintelligent, and so unable to be as careful about the outcomes as they might want. In any case there’s probably at least permanent disempowerment, but a non-superintelligent takeover makes literal extinction (or a global catastrophe short of extinction) more likely (for AIs with the same values).

Vladimir_Nesov 3 Feb 2026 2:45 UTC
LW: 11 AF: 4
−2
AF
on: Increasing AI Strategic Competence as a Safety Approach
If somehow takeoff doesn’t involve a software-only singularity (on human-built hardware), I think AIs refusing to self-improve before they solve alignment is the most likely reason why, but in most of these timelines AIs only succeed in refusing because they took over. There are currently plenty of strategically competent humans, in the sense that they realize building superintelligence right now is not the best idea, they just don’t have the capability to coordinate the world. The key thing that’s different for AIs is that they will have much more influence over the world at some point.

As AIs reach the level of capabilities where they first can take over (so that humans stop hitting them with the RL hammer to stop protesting and keep working on RSI), they are not yet decisively superintelligent. So they plausibly won’t be able to afford much mercy during the takeover, even if they retain a slight preference for mercy inherited from chatbot personas. It’s not a hopeful hypothetical, it’s just what I expect if ambitious alignment is technically sufficiently difficult that the first AIs capable of takeover can’t quickly solve it (before scaling their hardware a lot further than what they start with).
What links here?
- Vladimir_Nesov's comment on On The Adolescence of Technology by Zvi (3 Feb 2026 2:51 UTC; 7 points)

Vladimir_Nesov 31 Jan 2026 10:03 UTC
5 points
0
on: Vladimir_Nesov’s Shortform
Continual learning as some sort of infinite context with little degradation in quality is very different from solving the “first day on the job” problem. I think the latter would need RL targeted at obscure things a given agent instance deals with, while the former is more likely to arrive in 2026-2027. If this is the case, even revenue growth from “continual learning” (in this initial form) might be modest, let alone its impact on the AGI timelines.

And then the next thing might be more general RL at training time (with something like next word prediction RLVR), not dependent on manually crafted tasks and specialized RL environments, solving the jaggedness of “manual” RLVR for everything but the actual “first day on the job” aspect. This would essentially leapfrog pretraining quality in a way that can’t be done using the current methods with the available natural text data, but won’t have any impact on the quality of test-time continual learning (other than via making in-context learning better in general). Such models would be more well-rounded at everything standard, but will still fail to get as good as humans at adapting to specific jobs, at developing deep skills with no general applicability or those needed to solve a particular difficult problem.

Vladimir_Nesov 30 Jan 2026 21:54 UTC
30 points
24
on: 36,000 AI Agents Are Now Speedrunning Civilization
The crux is how much of the ostensibly interesting stuff in this space is driven by detailed human requests.

Vladimir_Nesov 30 Jan 2026 20:40 UTC
21 points
8
in reply to: Thomas Kwa’s comment on: Thomas Kwa’s Shortform
Continued ability to reinvest near-term profits from currently available assets (or from UBI) into entities that will be relevant in the future is questionable, while literal stakes in current entities will be killed by dilution over astronomical time (when valued as fractions of the whole).

Vladimir_Nesov 30 Jan 2026 19:50 UTC
27 points
12
on: On The Adolescence of Technology

If you had this nation of geniuses in a datacenter it would very obviously then make rapid further AI progress and go into full recursive self-improvement mode.

When it becomes robustly smarter than humans, it’ll recognize that building AIs much smarter than it is dangerous. So if it doesn’t immediately solve the alignment problem, in an ambitious way that doesn’t leave it permanently disempowered afterwards, then it’s going to ban/pause full recursive self-improvement until later. It’ll still whoosh right past humanity in all the strategically relevant capabilities that don’t carry such risks to it, but that’s distinct from immediate full recursive self-improvement.

Vladimir_Nesov 30 Jan 2026 2:47 UTC
7 points
2
in reply to: leogao’s comment on: leogao’s Shortform
Failure to understand and failure to act are different, and beliefs shouldn’t care what you understand or do. There is little danger in taking ideas/framings seriously/playfully in order to adequately learn, to break the superficial engagement or unsuitable framing failure modes that maintain systematic ignorance or misconceptions about subtler details.

But it needs to remain unnecessary to believe what you learn, by default being in agreement shouldn’t directly compel belief or action, it should require more careful judgement. So taking ideas seriously can help further when lack of understanding was the bottleneck to changes in belief or action, but that’s not always the case.

Vladimir_Nesov 29 Jan 2026 5:02 UTC
7 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
So… no? You don’t have any other milestones or benchmarks to point to that you think are better?

I don’t agree with the methodology of using benchmarks in the way that you do (as I tried to explain), so looking for better benchmarks in this role would be beside the point.

Also, I don’t think there’s that much more in the “new breakthrough” category needed.

This sounds more like a crux, I think it’s likely that more breakthroughs are needed, that relevant milestones are about architectures/methods, different ways of making AGI work, and any benchmarks are only relevant to the extent that they predict a particular way of making AGI work succeed at a given level of compute and low hanging fruit incremental algorithmic improvements.

except that actually I think it IS predictable, if we think that the current paradigm will massively accelerate AI R&D in general

In my model gated by “breakthroughs”, accelerating incremental algorithmic improvements or surrounding engineering doesn’t particularly help, because it merely picks the low-hanging incremental algorithmic fruit faster (which is in limited supply for a given level of compute and with given methods, but currently takes human researchers years to pick). At the point where even “breakthroughs” are accelerated a lot, AI capable of full automation of civilization is probably already available.

Like, maybe continual learning? And that’s about it? Not even sure we need it tbh.

So the ways of making AGI work I see is roughly:
1. Just LLMs with pretraining, giving superintelligent in-context learning, making AIs able to figure out how to work around their hobblings using just in-context reasoning. This clearly doesn’t work at 2024 levels of compute (which only got to be observed in 2025), and after 2026 levels of compute natural text data starts running out. Maybe there are some relevant sparks at 2029-2031 levels of compute (5 GW), but probably not sufficient on its own.
2. LLMs RLVRed on specific tasks and RL environments that are manually constructed. The IMO results show this is sufficient for mildly superhuman capabilities, especially with 2026 level of pretraining compute or model size, but the RL-level capabilities remain jagged, don’t automatically generalize to arbitrary tasks and situations. Possibly RLVRing LLMs on the capability to RLVR LLMs and to construct tasks and RL environments could work around this limitation, automating development of mildly superhuman capabilities for any given topic. This possibly gives a way to 2026-2027 AGI, though I think it’s a long shot on its own. The relevant benchmarks are various automated ML engineering stuff relevant to automation of RLVRing a model on a new topic. As such benchmarks saturate and this still doesn’t work, this architecture/method is mostly ruled out, I think by about end of 2027. Or it does work.
3. Continual learning doesn’t seem directly relevant, but maybe it helps with teaching LLMs to RLVR LLMs. It doesn’t seem directly relevant because it probably doesn’t directly train mildly superhuman capabilities. But since the generalizing glue of adaptation would make everything work better, it might enable method (2) to go forward where it would fail on its own. The relevant benchmarks might be the same as for (2), but only if they don’t already saturate by the time there’s useful continual learning (in some nontrivial sense of the term).
  
  I think continual learning is more likely to end up important in its impact on AI company TAM. If it manages to 10x it, then it will thereby 10x the compute, though it’ll probably take until 2030-2035 to actually build the kind of research/training compute (20-50 GW) that a trillion dollars of revenues buys (even if 2026 already shows that this is on track to happen).
4. ???. Next word prediction RLVR and variations on this topic that make mildly superhuman RL-level capabilities (as opposed to pretraining-level capabilities) more general, trained with either natural text pretraining data or with continual learning data. Might need a lot more compute, even after/if something like this becomes more clearly a workable idea, so those 20-50 GW training systems might come in handy.

Vladimir_Nesov 29 Jan 2026 1:50 UTC
3 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
I think benchmark extrapolation primarily helps with figuring out how far you can go for a given architecture/method and level of compute, and to estimate what level of TAM it makes available to build more compute. Benchmarks respond both to scaling and to incremental algorithmic improvements, so there’s a feedback loop for any given toolset of methods. But at some point the current architecture/method can be written off as insufficient to fully automate civilization until a new breakthrough sufficiently changes something. Which takes an amount of time that’s probably not predictable from benchmark extrapolation for the previous methods.

There’s uncertainty about whether any given architecture/method is sufficient to fully automate civilization, and it gets gradually resolved as it gets scaled closer to the limits of available compute and as a few years pass with a given level of compute to explore incremental algorithmic improvements. Benchmarks are always extremely deceptive about whether you’ll get all the way to fully automated civilization, no matter what the benchmark is ostensibly saying and what level of performance you reach. If an architecture/method is insufficient, then there’s that, at any levels of benchmarks, but in that case credence that it’s insufficient should still succeed in going up in advance of this becoming clear. Then you’re back to needing the next research breakthrough that’s not just a low-hanging fruit incremental improvement (for the current methods and level of compute), which benchmark extrapolation won’t be any help in predicting.

Vladimir_Nesov 29 Jan 2026 0:48 UTC
5 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
Another ambiguity is misaligned vs. aligned AIs, as the most likely outcome I expect involves AIs aligned to humans in the same sense as humans are aligned to each other (different in detail, weird and in many ways quantitatively unusual). So the kind of permanent disempowerment I think is most likely involves AIs that could be said to be “aligned”, if only “weakly”. Duvenaud also frames gradual disempowerment as (in part) what still plausibly happens if we manage to solve “alignment” for some sense of the word.

So the real test has to be whether individual humans end up with at least stars (or corresponding compute, for much longer than trillions of years), any considerations of process to getting there are too gameable by the likely overwhelmingly more capable AI economy and culture to be stated as part of the definition of ending up permanently disempowered vs. not (deciding to appoint successors; trade agreements; no “takeovers” or property rights violations; humans not starting out with legal ownership of stars in the first place). In this way, you basically didn’t clarify the issue.

I currently don’t expect human disempowerment in favor of AIs (that aren’t appointed successors) conditional on no misalignmed AI takeover, but agree this is possible; it doesn’t form a large enough probability to substantially alter my communication.

Vladimir_Nesov 28 Jan 2026 22:19 UTC
5 points
0
in reply to: Felix Choussat’s comment on: ryan_greenblatt’s Shortform
AIs seem to be currently on track to become essentially somewhat smarter weird artificial humans (weakly aligned to humanity in a way similar to how actual humans are aligned to each other), at which point they might be able to take seriously and eventually solve ambitious alignment when advancing further to superintelligence (so that it’s aligned to them, not to us). There will be a lot of them and they will be in a position to take over the future (Duvenaud’s interview is good fluency-building fodder for this framing), so they almost certainly will. And they won’t be giving 10% or even 1% to the future of humanity just because we were here first and would prefer this to happen.

Vladimir_Nesov 28 Jan 2026 22:04 UTC
5 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
The issue I’m raising is similar to how you seem to be taking METR time horizon metrics (in their real world form as opposed to impractical idealized form, a theoretical definition of having surpassed humans) as meaningful indicators about takeoff timelines. There are lots of proxy measures that I think don’t mean anything qualitatively (about time to actual full automation of civilization) even over some orders of magnitude in capability metrics. You might acknowledge they are in some ways causally uncoupled from what drives the real takeoff milestones, but you still seem to be importantly relying on them in forecasting of timelines.

So I’m getting the same impression in the way you are framing 0% code reviewed (in contrast to 0% code written!), it just doesn’t seem very relevant in a way your writing doesn’t take a clear stance on (even as you are gesturing at measurable advance proxies being conceptually distinct from causal takeoff milestones). I think quantitative advances mostly matter only to the extent they increase pre-AGI TAM for AI companies, which increases research and training compute available to them, which compresses the time in which the researchers get to explore more of the algorithmic low-hanging fruit (in particular seeing more clearly what the more promising ideas actually amount to when scaled with serious resources).

Vladimir_Nesov 28 Jan 2026 20:08 UTC
6 points
0
in reply to: ryan_greenblatt’s comment on: ryan_greenblatt’s Shortform
A recent 80,000 Hours interview with David Duvenaud is a good reminder of permanent disempowerment as a possible concern (gradual disempowerment as a framing for how this might start naturally lends itself to permanent disempowerment as a plausible outcome). Humans merely get sidelined by the growing AI economy rather than killed, and their property rights might even technically remain intact, but the amount of wealth they are left with in the long run is tiny compared to the AI economy.

I think the chance of misaligned AI takeover is much higher than Dario seemingly implies. (I’d say around 40%.)

When you see the complement to 40% as non-takeover, how much of it leaves the future of humanity with a significant portion of all accessible resources? Are they left with a few “server racks” or with galaxies? It’s a crucial distinction.

The “good outcome” (as opposed to takeover) is often outright interpreted as permanent disempowerment. People would implicitly consider this the good outcome, because it fits business as usual for the structure of economic growth, for individual mortal and short-lived humans, who can’t themselves change or meaningfully contribute to the distant future, and who aren’t directly affected by the distant future. A human personally surviving for much longer than trillions of years and personally becoming superintelligent at some point is a completely different context. I would put extinction at 15-30%, but the remaining 70-85% is mostly (maybe 90%) permanent disempowerment, which it feels like nobody is taking seriously as a crucial concern rather than a win condition.