John-Clark Levin

• John-Clark Levin 18 Feb 2026 8:42 UTC

  1 point

  0

  on: De­ci­sion the­ory does not im­ply that we get to have nice things

  Reality just contains vastly, vastly more compute than any of your sims can. Reality is the first world that has a compelling explanation of how the AI came to exist, and the first explanation where the resources that it uses to compute are continuous with the world that spawned it.

  I don’t understand why you seem to assume here that an intelligence could confidently infer anything about the computing conditions in a universe that’s simulating it.

  I do agree that our reality is vastly more self-consistent and complex than any faithful simulations we could make. I’m just challenging the claim that this lets us infer anything about the relationship between our reality and higher layers of hypothetical simulation.

  Reality is also the first world that can compellingly contain the signature of superintelligent optimization in it. For example, it’s plausibly very hard to fake the reaction of a real global stock market to some superintelligent trades without actually having a bunch of human brains running in the background. (And if this isn’t true of stock markets, it’s true of something.)

  I’m not clear on the logic underlying this example. Stock markets are affected by quantum-scale uncertainty bubbling up into macro-scale effects in domains like weather, war, and politics. Indeed, any real-world macro-scale system will be similarly affected. That level of noise would likely make it infeasible for a superintelligence to accurately judge its odds of being in a simulation from how its superintelligent actions affect the world.

  Further, a superintelligence wouldn’t have a firm foundation for inferring how much computation anything it observes would take. Distant galaxies could be simulated more efficiently by a planetarium, and when it sends a probe into the firmament the simulators could send back bogus data.

  And ultimately, the superintelligence could never be sure that it didn’t discover a smoking gun of the simulation 10 minutes ago, only for the simulators to swoop in Cartesian-evil-demon-style and edit its mind state to remove all trace of that.

  I agree it could reason its way toward regarding simulation scenarios as very unlikely on Solomonoff induction grounds, but the fundamental limitations above strike me as preventing the number of nines of certainty that you suggest elsewhere in the post.

• John-Clark Levin 15 Feb 2026 21:22 UTC

  11 points

  −1

  in reply to: Eli Tyre’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  Thanks, that’s a helpful clarification about your view, Eli! There are certainly scenarios where multiples that large could happen for certain investments. My qualification of EV terms factors in my all-things-considered view of the uncertainties—chiefly catastrophic risks and policies that distribute resources in some way other than “whoever owned the most stock when the growth curves accelerated”. Especially because the anti-democratic political dynamics that would make that outcome more likely would also in my view badly worsen catastrophic risk overall.

  As a further complication, the “S&P 500 stake → future galaxies” scenario seems to me to require an extremely fine-tuned Goldilocks level of power concentration: enough for our advanced spacefaring civilization to permanently lock billions of people out of meaningful shares in the future, but not so much that the people with even larger S&P 500 stakes at the Singularity can’t ever find a way to dispossess the people who cancelled their Italy trip and invested that $7,500 in Nvidia. I’d guess that even with maximal saving, most people reading this could only invest a 6-7 figure sum before AGI, which wouldn’t be a good counterweight to centibillionaires if democracy goes out the window.

  I’m not suggesting you hold quite such a stark version of that view, but that dynamic at least illustrates roughly why I consider it highly unlikely that the difference between, say, $100,000 invested and $107,500 or even $300,000 invested at takeoff will have any measurable impact on what share of resources the average person or their descendants can someday control outside this solar system.

• John-Clark Levin 14 Feb 2026 9:50 UTC

  1 point

  0

  in reply to: Eli Tyre’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  Yes, I roughly agree these would be valid counterexamples. My claim is that “usually” people in our age bracket don’t have things they can invest in that in EV terms will 1,000x by 2040 or 1,000,000,000x by 2050.

  Three related expectations I should make explicit:

  1) The basket of things to spend money on will be so different in 2040-2050 (most things getting significantly cheaper in real terms, many things becoming effectively free in real terms, and some things becoming much more expensive in real terms) that reasoning about the present value of future money faces much higher uncertainty than would have been true in 2006 about this decade. Most people’s consumption preferences tend to favor certainty, which pushes toward spending more on present experiences.

  2) Demand curves for most goods and services bending downward will tend to dominate overall deflation. There’s a certain regime when smart toilets getting cheaper increases sales, but once everyone upgrades all their toilets, further price cuts won’t induce people to buy even more. In the more general case, as goods and services become abundant, time will become the main bottleneck—as it already is for billionaires. This implies that, in a reversal of economic history so far, the real amount of money needed to satisfy the average person’s preferences would eventually decline.

  3) The impact of AGI will prompt policy shifts that limit the usefulness of capital. Sketching broadly, one can imagine a world where automation and abundance eventually severely diminish most people’s ability to trade labor for inherently scarce goods like Rembrandts or Malibu mansions. This would naturally lead to lock-in, where the people who own prestige property at the time of this economic transition keep it in perpetuity. There’s lots of uncertainty and contingency here, but I expect eventual political pressure to turn some kinds of prestige property into some kind of public good (e.g. a law that forbids private ownership of Old Masters or subjects it to conditions that limit the usefulness of ownership). I’m not sure quite how metaphorical you’re being about buying “fractional galaxies”, but I expect that policies will forbid that sort of thing.

  Finally, I’m not intending a fully general claim that you should be spending now instead of saving now. There are plenty of cases in which saving now gives you more flexibility in the near term as well. I’m referring rather more narrowly to thirtysomethings and twentysomethings saving for retirements that they imagine will be roughly like the world of 2026.

• John-Clark Levin 14 Feb 2026 7:57 UTC

  1 point

  0

  in reply to: xle’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  That is completely correct. To clarify in the light of the examples you give, my definition of spontaneity in the context of AI/​LLMs means specifically “action whose origin is unable to be traced back to the prompt or training data.” This is, sadly, difficult to prove as it would require proving a negative. I’ll give some thought to how I might frame this in such a way that it is verifiable in an immutable-goalpost kind of way but I’m afraid this isn’t something I have an answer for now. Perhaps you have some thoughts?

  I think that’s holding AI to a standard we don’t and can’t hold humans to. Every single thing you and I do that’s empirically measurable can plausibly be traced back in some way to our past experiences or observations—our training data. Spontaneity, desire, and emotion intuitively feel like a good bellwether of AGI consciousness because the sensations of volition and sentiment are so core to our experience of being human. But those aren’t strong cruxes of how much AGI would affect human civilization. We can imagine apocalyptically dangerous systems that design pandemic viruses without a shred of emotion, and likewise can imagine sublimely emotional and empathetic chatbots unable to either cause much harm or solve any real problems for us either. So I prefer the AGI definition I expressed largely because it avoids those murky consciousness questions and focuses on ability to impact the world in measurable ways.

  To continue your engine analogy, I think we can definitely agree that the “check engine” light is firmly on at this point. I think that the drawing a line in the sand for AGI vs. “very powerful LLM” is, at best, subjective, and distracts from the fact that the LLMs/​AIs that exist today are already well capable of causing the widescale damage that you warn of; the technology is already here, we are just waiting on the implementation. Perhaps what I mean is that we have, in my view, already crossed the line—the timing belt has snapped, the engine is dead, but we’re still coasting on the back of our existing momentum (maybe I’m over-stretching this analogy now...).

  We may have an object-level disagreement here. I agree that the “check engine” light is one, and that current AI can already cause many problems. But I also expect that there is a qualitative difference (again, not a bright line, though) between risk from today’s LLMs and from AGI. For example, current AI evals/​metrology have established to my satisfaction that the risk of GPT-5 class models designing an extinction level virus from scratch is extremely low.

  That’s a fair point, but if we aren’t arguing about “consciousness,” and we have grounded our definition of “AGI” in, essentially, its capacity to do damage, then I think these kinds of tests fall into the same category as GDP in economics: a reasonable corollary but ultimately unsuitable as a true metric (and almost certainly misleading and ripe for abuse if taken out of context).

  Absolutely, valid concerns. Folks in AI evals/​metrology are working very hard to make sure we’re measuring the right things, and to educate people about the limitations of those metrics.

  For sure! I just don’t feel the need to wait for this technology to be relabeled as “AGI” before we do something about it. If your concern is their ability to act, as the agents on Moltbook act, (let’s say) “semi-spontaneously,” then we are clearly already there: all we are waiting for is for a person to hand over the launch codes to an agent (or put a crowd of them in charge of a social-media psy-op, prior to a key election, etc.).

  Yes, I am not suggesting that we wait. We should be acting aggressively now to mitigate risks.

  You say that AIs would need to be “qualitatively” different to current generation models to do pose enough of a threat to be worthy of the “AGI” label. Please could you outline what these qualitative differences might be? I can only think of quantitative differences (e.g. more agents, more data-centers, more compute, more power, wider-scale application/​deployment, more trust, more training data—all of these are simply scaling-up what already is and require no truly novel technology, though they would all increase the risk posed by AIs to our society).

  The qualitative differences I’m referring to often involve threshold effects, where capabilities above the threshold trigger different dynamics. Sort of like how the behavior of a 51 kg sphere of enriched uranium is a very poor guide to the behavior of a 52 kg sphere at critical mass. Some concrete examples include virus design (synthesizing a high-lethality virus with

  

  is a pandemic, and lower than that generally isn’t), geoengineering (designing systems capable of triggering climatic chain reactions, such as superefficient carbon-capture algae), nanotechnology (designing nanobots that can self-replicate from materials common in the biosphere). In all those cases, the dynamics of a disaster would be wildly different from an AI malfunction at lower levels of capability.

  As for your point that you, personally, are concentrating on the individual response within the wider community of alarmists who, collectively, are concentrating on both the collective and the individual response: thank you for clarifying this, it is important context. I definitely agree that both avenues need exploration and it is no bad thing to concentrate your efforts. I would say that, for my rope, the collective response is where I think the overall course will be set, but when collectivism fails, then individualism (or, more realistically, smaller scale collectivism) is the backstop. In this vein, I think that point 10 from your original article is the absolute key: it won’t be your basement full of tinned food that saves you from the apocalypse: it will be your neighbours.

  Perhaps I worded this in an unclear way. I am personally concentrating mostly on the collective response. But this particular post is about the individual response, partly because there is less clear and accessible material about that than on the collective response, which is a major focus of many other LessWrong posts.

  Many thanks for the thoughtful exchange!

• John-Clark Levin 12 Feb 2026 3:58 UTC

  2 points

  1

  in reply to: xle’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  Very helpful amplifications, xle! Much appreciated.

  I really do get the appeal of the “spontaneously express … complex desire or emotion” framing, but if I’m understanding you correctly, the whole thing basically hinges “spontaneous”, since AI can already express complex desires and emotions when we prompt it to. But agents on Moltbook are already expressing what purport to be complex desires and emotions even without any prompting. If this doesn’t count because the agents were first instructed to go do things spontaneously, we start to see that “spontaneous” is a very slippery thing to define. Ultimately, any action of an AI we create can be traced back to us, so is in some sense not spontaneous. So it’s worth thinking as concretely as you can about how you’d define spontaneity clearly enough that it could be proven by a future scientific experiment, and in a way that would resist post hoc goalpost-moving by skeptics.

  Your “90th percentile” operationalization is a good way of getting at roughly the AGI definition I’m endorsing. One issue to flag, though. AGI will have massive impacts, and it will be important to have some warning. If the minimal thing that would increase your credence of “AGI soonish” is AGI itself, you’d be committing yourself to not having any warning. Yes, the engine sputtering and dying is a very solid signal that you’re out of gas, but also a very costly and dangerous signal. So there’s value in figuring out your equivalent of a fuel gauge warning that lights up while the engine is still running fine—something pre-AGI that would convince you that AGI is probably coming soon.

  What I’m getting at about mathematics is just that it’s a domain that’s effectively independent of human culture, so not subject to anthropomorphization in the way that writing haikus or saying “I love you” is.

  I agree that who holds the proverbial launch codes is of extraordinary importance, and that we must marshal enormous civilization-level effort toward governing AGI responsibly, justly, and safely. That is, in fact, a much more central concern of my research than the subject of this post, which is individual-level preparedness. We absolutely need both. But I am making the additional claim that AGI will have the capacity to act with meaningful agency—to decide on targets and launch itself, in the nuclear weapons analogy—and that this introduces a qualitatively different set of challenges above and beyond the political ones. I don’t intend it as an absolute line in the sand between AGI and today’s LLMs, but I do claim that qualitative difference to be very important.

  It’s good to see on how much we’ve come to agree on here, despite approaching this with different framings.

• John-Clark Levin 11 Feb 2026 2:55 UTC

  2 points

  1

  in reply to: xle’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  Thanks — glad you found that helpful! That’s a good clarification. One thing I invite you to consider: what is the least impressive thing that AI would need to significantly increase your credence in AGI soonish?

  To clarify, the definition of AGI I’m using (AI at least at the level of educated humans across all empirically measurable cognitive tasks) does not entail any claims about true consciousness. It’s narrowly a question about functional performance.

  I think AI progress in very pure fields like mathematics is our best evidence that this isn’t an anthropomorphic illusion—that AI is actually doing roughly the same information-theoretic thing that our brains are doing.

  Your outside-view skepticism of doom scenarios is certainly warranted. My counterargument is: should a rational person have dismissed risks of nuclear annihilation for the same reason? I claim no, because the concrete inside-view reasons for considering doom plausible (e.g. modeling of warhead yields) were strong enough to outweigh an appropriate amount of skepticism. Likewise, I think the confluence of theoretical reasons (e.g. instrumental convergence) and empirical evidence (e.g. alignment faking results) are strong enough to warrant at the very least some significant credence in risks of doom.

• John-Clark Levin 10 Feb 2026 19:18 UTC

  1 point

  0

  in reply to: cubefox’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  I agree about overall deflation, and relative exceptions for land/​housing barring policy interventions.

• John-Clark Levin 10 Feb 2026 3:48 UTC

  2 points

  0

  in reply to: xle’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  Thank you! That’s an enormous topic that many other posts here have treated in more depth than I could hope to in this comment, but I’ll broadly gesture at a few key reasons why I believe AGI is probable (>50% before 2030, and >80% before 2037).

  • As of 2026, AI has already replicated most of human intelligence, including highly flexible capabilities like language use and zero-shot in-context reasoning. There are only a few big milestones between here and AGI, which have become much better theorized in the open literature than they were even two years ago. Frontier labs now have a small shopping list of capabilities like world modeling and continuous learning that they need to crack, and are applying Apollo Program-scale resources toward doing so.

  • Although these remaining problems are very hard, none of them appear totally unyielding in the way previous bottlenecks did. Before PaLM, for example, AI scientists were looking ahead at what we now call chain-of-thought, and it seemed like a towering black cliff face rearing up ahead, and nobody had any pitons or rope for the climb. There was almost zero progress on problems requiring chain-of-thought for years. Today’s models already do mediocre world modeling and there are a few different approaches giving us some purchase on continuous learning.

  • There are now several lines of empirical evidence converging on short AGI timelines. From Kurzweil (1999) through Cotra (2020), major AGI predictions were exclusively theory-derived—predicting future AI performance based not on current performance trends but the hypothesis that neural networks were the most promising path to AGI, and that a combination of compute cost trends and assumptions about the needed scale of compute could predict when we’d get AGI. We now have more evidence for that case too, with steady exponential gains in not just computing hardware price-performance but also algorithmic efficiency and compute scale. But more importantly, we now have strong empirics on AI capabilities progress itself and detailed quantitative modeling of how automated coding speeds up AI progress, and direct AI performance metrics like completing long time horizon tasks.

  • Yes, any progress curve could suddenly stop. But when a curve has held steady for long enough, that’s not the way to bet. Computation price-performance has already marched through 17 orders of magnitude since 1939. And for almost that entire time, engineers felt they were near the very limit of what was feasible. We’ve already covered most of the capabilities ground between Attention Is All You Need (2017) and AGI, so absent evidence to the contrary (which we haven’t seen yet) our priors should be weighted toward progress continuing at least that far again.

  • Humans are the existence proof. And there is massive headroom (several orders of magnitude, depending on how you frame the question) for deep learning to improve its sample efficiency and energy efficiency. Current ML techniques are nowhere near information-theoretic limits. That we’ve already gotten such progress with very “vanilla” statistical methods is evidence that there’s a lot of juice left to be squeezed.

  What do you see as the strongest reasons for considering AGI improbable?

• John-Clark Levin 10 Feb 2026 2:51 UTC

  1 point

  −2

  in reply to: cubefox’s comment on: Eleven Prac­ti­cal Ways to Pre­pare for AGI

  Everyone’s circumstances vary, but I expect for most people reading LW, there won’t be enough time between now and AGI to accumulate sufficient capital to live off for the rest of their lives if their labor value reaches zero.

  That said, I do endorse saving some emergency funds for overall resilience.

Eleven Prac­ti­cal Ways to Pre­pare for AGI