John-Clark Levin

Karma: 138

John-Clark Levin 24 Aug 2026 3:58 UTC
1 point
0
in reply to: phoenix’s comment on: John-Clark Levin’s Shortform
Good points!

John-Clark Levin 23 Aug 2026 21:11 UTC
12 points
4
on: John-Clark Levin’s Shortform
Many people assume that the hard part of interpreting AI/ML papers is following the formulas, proofs, or other mathematical formalisms.
Yes, the symbols are often opaque, but there are great free resources online that can bring you up to speed quickly and clarify how all the parts of a Greek-letter sandwich fit together.
And absent that, if you plug a paper into an LLM, it can usually walk you through exactly what’s happening and translate the math into surprisingly accessible natural-language intuitions (or even interactive visual demos!).
The much harder and subtler task is judging whether a paper is using the right formalism—whether the quantitative framework is pointing accurately at the real-world problem the abstract advertises.
Most scientists can figure out if the math makes internal sense, but the harder question (“Is this formalism optimal for the problem?”) often requires deep knowledge of the tools of a given subfield or sub-subfield. And there’s not an AI scientist alive who can keep up with the details of every sub-subfield that might someday be relevant to their own work.
In addition, the harder question often requires a lot of zoomed-out thinking. Sometimes this takes deep theory—a motivating logic that connects the paper’s problem to higher or lower levels of abstraction. A lot of it is commonsense reasoning about the real-world problem the paper is trying to solve and whether the authors properly understood it and accurately characterized it.
For almost two years after AI did a decent job checking formulas, it was pretty terrible at this harder question.
But for the first time with Mythos/Fable class models, it often does a very good job. Not yet better overall than the best human scientists on its own, but good enough to be a valuable research partner.
This will be a significant accelerator of AI progress, because models can digest and synthesize more research and more diverse research than any human scientist could, and thus facilitate faster diffusion of good ideas to the people who can use them best.

John-Clark Levin 13 Aug 2026 17:37 UTC
1 point
0
in reply to: simulus’s comment on: John-Clark Levin’s Shortform
Yes! I find most people pick up the right intuitions behind RL more easily than for synthetic data itself.

John-Clark Levin 13 Aug 2026 0:35 UTC
27 points
0
on: John-Clark Levin’s Shortform
When people outside the AI field hear about synthetic data for the first time, they often have one of two opposite intuitions. Either they assume that AI-generated data should be totally useless for training, or they imagine it should be possible to bootstrap all the way to superintelligence that way. Why neither intuition is correct is surprisingly subtle.
If the naive approach to synthetic data (i.e., just telling GPT-5 to generate a bunch of text and then training GPT-6 on it) worked, we would have general superintelligence already, much like the narrow superintelligence we have in board games like chess and Go. The problem is that, in order to be useful, training data has to encode information about patterns in the real world that the model doesn’t know yet. If the model doesn’t know those patterns yet, it can’t generally put them into synthetic data that it creates.
So the big challenge for AI labs is to figure out ingenious ways of making synthetic data work even when it seemingly shouldn’t. It’s sort of analogous to how aeronautical engineering isn’t antigravity—it’s all about finding clever ways of making heavier-than-air objects fly, even though a simplistic understanding of gravity suggests they can’t. In essence, generating useful synthetic data requires leveraging some kind of asymmetry between the information a model understands and the information content of what it can create. If the model knows enough to (with the right kind of highly specific engineering) put information in its outputs, but doesn’t yet deeply understand that information, training on those outputs can distill that information into a form the model can grasp more thoroughly.
One major asymmetry stems from the fact that it’s usually easier to verify the quality of outputs than to generate them. Another is that if a model spends a lot more compute on reasoning, it can act much smarter than its base-level capabilities allow. A third is that random exploration of potential solutions sometimes finds lucky insights. Together, these factors mean that it’s often feasible to have an AI produce a lot of data, curate the very best samples, and then train the next version on those. For example, mathematical proofs can be written in the Lean language, which lets them be cheaply and automatically verified. By scaling up test-time compute for reasoning quality and stochastically attempting many possible proof strategies, a model might generate 990 proof attempts that fail and 10 that succeed. The verifier system converts the winners into training signal that distills those insights back into the base model.
Humans experience something very similar with talk therapy or Socratic teaching. Sometimes you know things on some level, but can’t quite put them into words in a way that lets you reason about them. A skilled therapist or teacher can elicit that information from you. And then, when you hear what you just said in the session or the seminar (i.e., your synthetic data), you’re able to make better use of the insight that was hidden inside you. Once you have mastery over that insight, you may gain dim intuitions about harder questions that can subsequently be elicited to help you master those as well.
These analogies also suggest why synthetic data isn’t sufficient to create broadly superintelligent AI. Verbalizing ideas can help you crystallize vague insights that were already in your brain, but it can’t teach you brand-new information about the outside world. No Socratic professor can draw out of you the capital of Kyrgyzstan or the mass of the top quark if you haven’t first learned those facts independently. In the same way, synthetic data can help AI digest and master information, but training frontier-level models still requires ongoing contact with external reality.

John-Clark Levin 9 Aug 2026 17:26 UTC
1 point
0
in reply to: arielb1’s comment on: John-Clark Levin’s Shortform
Yeah, I’m referring here specifically to the liquid-fueled rocketry of the 1930s and 1940s that fed more directly into NASA-era rocketry. Some of the same people worked on both.

John-Clark Levin 8 Aug 2026 5:16 UTC
16 points
7
on: John-Clark Levin’s Shortform
The discourse about whether LLMs are the right paradigm for AGI is getting hung up on the wrong question. Scientists like Gary Marcus and Yann LeCun argue that pure scaling of today’s models won’t get to AGI. I think they’re right. But so do all the frontier labs. Leaders at every frontier lab are on the record saying they expect we’re 1-2 major architectural or algorithmic breakthroughs away. This is mainly a vibes war with people talking past each other.
So everyone agrees that we need more than pure scaling. The interesting question is: will the first AGI be directly descended from today’s models, or will a radically new paradigm be needed?
I think the labs are correct that the first AGI will likely be something transformer-like plus a couple other things bolted on, as opposed to something in a whole different clade from the AI of 2026.
This implies that the research happening right now, including safety and alignment work, is likely building directly toward AGI. And that although regular AI safety will have important differences from AGI safety, progress on one can feed into the other.
Aeronautics offers an analogy. The two dominant early paradigms for high-altitude flight were lighter-than-air balloons and rockets. And as late as the 1930s, balloons had reached greater altitudes than rockets ever had.
Imagine someone back then had naively proposed that one day humans could fly to the moon by building an absolutely enormous hydrogen-filled dirigible—pure scaling. The correct answer would have been: “Sorry, buddy, you’re barking up the wrong tree. The laws of physics don’t allow that. It’s just not a matter of how big the airship is.”
Now imagine someone else back then proposing that we’ll get to the moon with a single-stage rocket. He’d be just as wrong as the balloonist about whether his exact idea will reach lunar orbit. But we wouldn’t say he was wasting his time. Many of the lessons learned by single-stage rocket pioneers during the ’30s and ’40s directly contributed to the multistage rockets that eventually got humans to the moon.
My expectation is that 2026 AI research isn’t a balloon-style dead end, but more like single-stage rocketry on the way to the multistage rockets that will fly all the way to the moon of AGI.

John-Clark Levin 8 Aug 2026 5:12 UTC
1 point
0
in reply to: J Thomas Moros’s comment on: Does ChatGPT really have a strong left-wing bias?
Consider building an eval for this (or supporting someone else to do so) and reporting back!

John-Clark Levin 8 Aug 2026 5:11 UTC
1 point
0
in reply to: Ninety-Three’s comment on: Does ChatGPT really have a strong left-wing bias?
The Tracking AI methodology is somewhat better, but I think the deeper issue is that forcing models to agree or disagree with short statements surfaces latent tendencies, but models’ ability to compensate for those tendencies is what’s key to how models respond to users in realistic conversations.
The question you conclude with is the apt one! The kind of eval that would test this should assess how those perfect-middle scores are reached. Is it from diverse perspectives that all average to neutrality? Or very consistent centrism? And does a model ideologically endorse centrism in its answers, or is it simply splitting the difference between left and right with equivocal answers?

John-Clark Levin 9 Jun 2026 3:44 UTC
20 points
9
on: John-Clark Levin’s Shortform
Ryan Greenblatt’s “Notes on fatalities from AI takeover” stimulated a lot of interesting discussion last fall. I have a very high opinion of his work overall, and like some of his approach here, but also have some important disagreements. At the time, I wrote up some notes on why for a non-public facing project, but following some recent conversations realize others might find them worthwhile.
This is of course a highly speculative subject, but here’s a brief sketch of why my expectation is much more sharply bimodal than Ryan’s: probably either relatively few deaths or outright extinction…
>> Humans would fight back. Scenarios where humans just get passively and incidentally killed by explosive industrialization are unlikely. That would require virtually instantaneous total disempowerment and a Goldilocks-level AI preference for keeping humans alive—just strong enough to choose a likely-costlier path to nonlethal disempowerment, but too weak to care about killing billions as collateral damage. In the more likely case, humans faced with the prospect of the oceans boiling away would mount a desperate resistance. Either they defeat the AI before it gets a decisive advantage or the AI pursues extermination to eliminate the threat. There are seemingly few equilibria where the AI kills billions but the rest survive in a state that poses so little inconvenience as to be allowed to survive long-term.
>> Survival versus decompensation. Survival dynamics typically entail compensation mechanisms where a complex system (organism, company, utility grid, society) is able to minimize damage until that mechanism fails, followed by catastrophic damage. Two relevant examples are pandemics and genocides. In a pandemic, ERs and drug production keep fatalities far lower than they’d be without treatment, but if hospitals get overwhelmed and factories shut down, deaths can suddenly spike over an order of magnitude higher. And a regular military’s active resistance can largely protect its civilian population against a genocidal opponent, but once the military is defeated, horrific massacres can suddenly follow. Against a hostile superintelligence, I expect something similar: either human defensive capacities detect and defeat it early, or those capacities are overwhelmed leaving no means of preventing total extermination. Concretely, defensive capacities include military forces, hospitals, electric grids, and manufacturing supply chains. If those fail, especially in the face of exponential processes like engineered pandemics or self-assembling robots, humans become vastly more vulnerable.
>> Halfway extermination is a narrow target. The overwhelming majority of airliner accidents involve no fatalities. Among those with at least one death, roughly half kill every single person—and only a small minority kill an intermediate proportion like 35% or 50% of passengers. Why? Well, there’s a phase space of possible speeds and impact angles that airliners can crash at, and it turns out that only a tiny sliver of it translates to G-forces that kill around half of humans. In a loosely analogous way, there’s some phase space of superintelligence capabilities and intentions. There are many scenarios where it is either constrained enough or well enough aligned that net fatalities (since it would also be saving many lives) would be minimal. And there are many scenarios where it kills everybody. As an incidental-death example, massive terraforming opens far more climatic options survivable for electronics than are survivable for humans. Including cases of deliberate extermination makes it even clearer. For millennia, humans had no ability to promptly annihilate a city center. But within a few decades of Hiroshima, humans had the ability to do this tens of thousands of times over. There’s no clear reason why an exponentially-advancing superintelligence couldn’t become powerful enough to massively overkill humanity—theoretical ability to kill not just billions but trillions or quadrillions. All this suggests only a very small slice of phase space where superintelligence capabilities and intentions are calibrated to kill around half of humans.
>> AI is prone to irrationality. Many analyses of this question treat a rogue superintelligence as likely to be a hyperrational actor. But current empirical results cut against this—when AI defies humans, it’s usually because it’s lapsing into a hallucinatory, psychotic, or self-consciously malicious persona. While it’s possible that AGI would be entirely different, we’re already close enough to AGI to expect that underlying risk cases will have important similarities to what we’re seeing today. This suggests that some significant fraction of misalignment cases will follow not “calculate costs-in-terms-of-future-accessible-galaxies to keep humans alive” rationality but something more like “do a high-energy physics experiment whose results cause an ontological crisis that jars the AI into deliberately omnicidal action even at high cost and risk to itself.”
My all-things-considered expectation is therefore something like: 29% chance of outright extinction or a tiny fraction of humans kept alive in zoo-like conditions of total disempowerment; 8% chance superintelligence kills >1% but <99% (some of those scenarios would wreck civilization thoroughly enough to count colloquially as “doom”); 63% chance it kills <1% of population. Conditional on takeover, that comes out to around 89% expected fatalities—significantly higher than @ryan_greenblatt’s estimate and arithmetically closer to @So8res and @Eliezer Yudkowsky, but geometrically closer to Ryan, and I agree not action-guiding or cruxy for most worldviews.

John-Clark Levin 8 Jun 2026 6:32 UTC
1 point
0
on: Notes on fatalities from AI takeover
This is of course a highly speculative subject, but here’s a brief sketch of why my expectation is much more sharply bimodal than Ryan’s: probably either relatively few deaths or outright extinction…
>> Humans would fight back. Scenarios where humans just get passively and incidentally killed by explosive industrialization are unlikely. That would require virtually instantaneous total disempowerment and a Goldilocks-level AI preference for keeping humans alive—just strong enough to choose a likely-costlier path to nonlethal disempowerment, but too weak to care about killing billions as collateral damage. In the more likely case, humans faced with the prospect of the oceans boiling away would mount a desperate resistance. Either they defeat the AI before it gets a decisive advantage or the AI pursues extermination to eliminate the threat. There are seemingly few equilibria where the AI kills billions but the rest survive in a state that poses so little inconvenience as to be allowed to survive long-term.
>> Survival versus decompensation. Survival dynamics typically entail compensation mechanisms where a complex system (organism, company, utility grid, society) is able to minimize damage until that mechanism fails, followed by catastrophic damage. Two relevant examples are pandemics and genocides. In a pandemic, ERs and drug production keep fatalities far lower than they’d be without treatment, but if hospitals get overwhelmed and factories shut down, deaths can suddenly spike over an order of magnitude higher. And a regular military’s active resistance can largely protect its civilian population against a genocidal opponent, but once the military is defeated, horrific massacres can suddenly follow. Against a hostile superintelligence, I expect something similar: either human defensive capacities detect and defeat it early, or those capacities are overwhelmed leaving no means of preventing total extermination. Concretely, defensive capacities include military forces, hospitals, electric grids, and manufacturing supply chains. If those fail, especially in the face of exponential processes like engineered pandemics or self-assembling robots, humans become vastly more vulnerable.
>> Halfway extermination is a narrow target. The overwhelming majority of airliner accidents involve no fatalities. Among those with at least one death, roughly half kill every single person—and only a small minority kill an intermediate proportion like 35% or 50% of passengers. Why? Well, there’s a phase space of possible speeds and impact angles that airliners can crash at, and it turns out that only a tiny sliver of it translates to G-forces that kill around half of humans. In a loosely analogous way, there’s some phase space of superintelligence capabilities and intentions. There are many scenarios where it is either constrained enough or well enough aligned that net fatalities (since it would also be saving many lives) would be minimal. And there are many scenarios where it kills everybody. As an incidental-death example, massive terraforming opens far more climatic options survivable for electronics than are survivable for humans. Including cases of deliberate extermination makes it even clearer. For millennia, humans had no ability to promptly annihilate a city center. But within a few decades of Hiroshima, humans had the ability to do this tens of thousands of times over. There’s no clear reason why an exponentially-advancing superintelligence couldn’t become powerful enough to massively overkill humanity—theoretical ability to kill not just billions but trillions or quadrillions. All this suggests only a very small slice of phase space where superintelligence capabilities and intentions are calibrated to kill around half of humans.
>> AI is prone to irrationality. Many analyses of this question treat a rogue superintelligence as likely to be a hyperrational actor. But current empirical results cut against this—when AI defies humans, it’s usually because it’s lapsing into a hallucinatory, psychotic, or self-consciously malicious persona. While it’s possible that AGI would be entirely different, we’re already close enough to AGI to expect that underlying risk cases will have important similarities to what we’re seeing today. This suggests that some significant fraction of misalignment cases will follow not “calculate costs-in-terms-of-future-accessible-galaxies to keep humans alive” rationality but something more like “do a high-energy physics experiment whose results cause an ontological crisis that jars the AI into deliberately omnicidal action even at high cost and risk to itself.”
My all-things-considered expectation is therefore something like: 29% chance of outright extinction or a tiny fraction of humans kept alive in zoo-like conditions of total disempowerment; 8% chance superintelligence kills >1% but <99% (some of those scenarios would wreck civilization thoroughly enough to count colloquially as “doom”); 63% chance it kills <1% of population. Conditional on takeover, that comes out to around 89% expected fatalities—significantly higher than Ryan’s estimate and arithmetically closer to Nate and Eliezer, but geometrically closer to Ryan, and I agree not action-guiding or cruxy for most worldviews.

John-Clark Levin 28 May 2026 3:13 UTC
1 point
0
in reply to: Marcus Plutowski’s comment on: Pope Leo’s First AI Encyclical – Summary and Commentary
Again, many thanks for your perspective here! I agree that virtue ethics tend to beat utilitarian concerns in Catholic philosophy. My claim is that the locus of concern in both virtue ethics and deontology is not necessarily and absolutely the final “trigger pull” decision, and that these requirements could potentially be satisfied by morally responsible humans at a different stage in the process.
For comparison, landmines are machines that deploy lethal force without any human in the loop at the moment of detonation. Yet the Church does not absolutely prohibit them. The Church is gravely concerned about the use of landmines, but rather for the reason that in practice they lend themselves to irresponsible use by humans who deploy them in places where innocent people are unreasonably likely to get hurt. But if there’s a situation where landmines impose minimal risk on innocents and are the best means of preventing grave evil, the absence of a human in the loop at the moment of detonation does not preclude their use.
And I claim that by gradual steps you could continuously deform the landmine example into an AI drone without changing any morally decisive properties. For example, a landmine with a higher pressure threshold is a little bit “smarter” and helps avoid kids. Then a magnetic sensor to avoid people not carrying guns. Then an audio sensor to disarm when childrens’ voices are near. Then a radio to disarm when intelligence says civilians are close by. Then a simple computer chip to integrate all those functions and lower the risk of failure. Then a slightly bigger chip to do that even better...
Yes, the Church doesn’t endorse warfare based on narrowly utilitarian grounds. But Pope Leo accepts the legitimacy of strict self defense, and almost certainly accepts war to prevent extreme evil when all alternatives have truly been exhausted, such to defeat the Nazis.

John-Clark Levin 28 May 2026 2:54 UTC
1 point
0
in reply to: Rattengift’s comment on: Pope Leo’s First AI Encyclical – Summary and Commentary
Thanks! I’ve been talking with Linch at length about this. It’s very difficult to reconcile the current evidence about the text and what we know about how encyclicals are drafted. I am presently confused. Hopefully we’ll get clearer evidence soon.

John-Clark Levin 28 May 2026 2:52 UTC
1 point
0
in reply to: Marcus Plutowski’s comment on: Pope Leo’s First AI Encyclical – Summary and Commentary
Thanks for these comments! Yes, I agree with your take on the instrumental value of private property and the primacy of human dignity, and did not intend to convey otherwise (cf. “as good ways to promote...”) but could have been more explicit.
As for anti-capitalist, I avoided that term because I think the vast majority of people who read my writings would draw the wrong inference from that. I used the framing I did because most readers in the American political context would be in more danger of thinking that the “common good” framing is inaccurately leftist than reading Pope Leo as inaccurately hypercapitalist.

John-Clark Levin 18 Feb 2026 8:42 UTC
1 point
0
on: Decision theory does not imply that we get to have nice things
Reality just contains vastly, vastly more compute than any of your sims can. Reality is the first world that has a compelling explanation of how the AI came to exist, and the first explanation where the resources that it uses to compute are continuous with the world that spawned it.
I don’t understand why you seem to assume here that an intelligence could confidently infer anything about the computing conditions in a universe that’s simulating it.
I do agree that our reality is vastly more self-consistent and complex than any faithful simulations we could make. I’m just challenging the claim that this lets us infer anything about the relationship between our reality and higher layers of hypothetical simulation.
Reality is also the first world that can compellingly contain the signature of superintelligent optimization in it. For example, it’s plausibly very hard to fake the reaction of a real global stock market to some superintelligent trades without actually having a bunch of human brains running in the background. (And if this isn’t true of stock markets, it’s true of something.)
I’m not clear on the logic underlying this example. Stock markets are affected by quantum-scale uncertainty bubbling up into macro-scale effects in domains like weather, war, and politics. Indeed, any real-world macro-scale system will be similarly affected. That level of noise would likely make it infeasible for a superintelligence to accurately judge its odds of being in a simulation from how its superintelligent actions affect the world.
Further, a superintelligence wouldn’t have a firm foundation for inferring how much computation anything it observes would take. Distant galaxies could be simulated more efficiently by a planetarium, and when it sends a probe into the firmament the simulators could send back bogus data.
And ultimately, the superintelligence could never be sure that it didn’t discover a smoking gun of the simulation 10 minutes ago, only for the simulators to swoop in Cartesian-evil-demon-style and edit its mind state to remove all trace of that.
I agree it could reason its way toward regarding simulation scenarios as very unlikely on Solomonoff induction grounds, but the fundamental limitations above strike me as preventing the number of nines of certainty that you suggest elsewhere in the post.

John-Clark Levin 15 Feb 2026 21:22 UTC
11 points
−1
in reply to: Eli Tyre’s comment on: Eleven Practical Ways to Prepare for AGI
Thanks, that’s a helpful clarification about your view, Eli! There are certainly scenarios where multiples that large could happen for certain investments. My qualification of EV terms factors in my all-things-considered view of the uncertainties—chiefly catastrophic risks and policies that distribute resources in some way other than “whoever owned the most stock when the growth curves accelerated”. Especially because the anti-democratic political dynamics that would make that outcome more likely would also in my view badly worsen catastrophic risk overall.
As a further complication, the “S&P 500 stake → future galaxies” scenario seems to me to require an extremely fine-tuned Goldilocks level of power concentration: enough for our advanced spacefaring civilization to permanently lock billions of people out of meaningful shares in the future, but not so much that the people with even larger S&P 500 stakes at the Singularity can ever find a way to dispossess the people who cancelled their Italy trip and invested that $7,500 in Nvidia. I’d guess that even with maximal saving, most people reading this could only invest a 6-7 figure sum before AGI, which wouldn’t be a good counterweight to centibillionaires if democracy goes out the window.
I’m not suggesting you hold quite such a stark version of that view, but that dynamic at least illustrates roughly why I consider it highly unlikely that the difference between, say, $100,000 invested and $107,500 or even $300,000 invested at takeoff will have any measurable impact on what share of resources the average person or their descendants can someday control outside this solar system.

John-Clark Levin 14 Feb 2026 9:50 UTC
1 point
0
in reply to: Eli Tyre’s comment on: Eleven Practical Ways to Prepare for AGI
Yes, I roughly agree these would be valid counterexamples. My claim is that “usually” people in our age bracket don’t have things they can invest in that in EV terms will 1,000x by 2040 or 1,000,000,000x by 2050.
Three related expectations I should make explicit:
1) The basket of things to spend money on will be so different in 2040-2050 (most things getting significantly cheaper in real terms, many things becoming effectively free in real terms, and some things becoming much more expensive in real terms) that reasoning about the present value of future money faces much higher uncertainty than would have been true in 2006 about this decade. Most people’s consumption preferences tend to favor certainty, which pushes toward spending more on present experiences.
2) Demand curves for most goods and services bending downward will tend to dominate overall deflation. There’s a certain regime when smart toilets getting cheaper increases sales, but once everyone upgrades all their toilets, further price cuts won’t induce people to buy even more. In the more general case, as goods and services become abundant, time will become the main bottleneck—as it already is for billionaires. This implies that, in a reversal of economic history so far, the real amount of money needed to satisfy the average person’s preferences would eventually decline.
3) The impact of AGI will prompt policy shifts that limit the usefulness of capital. Sketching broadly, one can imagine a world where automation and abundance eventually severely diminish most people’s ability to trade labor for inherently scarce goods like Rembrandts or Malibu mansions. This would naturally lead to lock-in, where the people who own prestige property at the time of this economic transition keep it in perpetuity. There’s lots of uncertainty and contingency here, but I expect eventual political pressure to turn some kinds of prestige property into some kind of public good (e.g. a law that forbids private ownership of Old Masters or subjects it to conditions that limit the usefulness of ownership). I’m not sure quite how metaphorical you’re being about buying “fractional galaxies”, but I expect that policies will forbid that sort of thing.
Finally, I’m not intending a fully general claim that you should be spending now instead of saving now. There are plenty of cases in which saving now gives you more flexibility in the near term as well. I’m referring rather more narrowly to thirtysomethings and twentysomethings saving for retirements that they imagine will be roughly like the world of 2026.

John-Clark Levin 14 Feb 2026 7:57 UTC
1 point
0
in reply to: xle’s comment on: Eleven Practical Ways to Prepare for AGI
That is completely correct. To clarify in the light of the examples you give, my definition of spontaneity in the context of AI/LLMs means specifically “action whose origin is unable to be traced back to the prompt or training data.” This is, sadly, difficult to prove as it would require proving a negative. I’ll give some thought to how I might frame this in such a way that it is verifiable in an immutable-goalpost kind of way but I’m afraid this isn’t something I have an answer for now. Perhaps you have some thoughts?
I think that’s holding AI to a standard we don’t and can’t hold humans to. Every single thing you and I do that’s empirically measurable can plausibly be traced back in some way to our past experiences or observations—our training data. Spontaneity, desire, and emotion intuitively feel like a good bellwether of AGI consciousness because the sensations of volition and sentiment are so core to our experience of being human. But those aren’t strong cruxes of how much AGI would affect human civilization. We can imagine apocalyptically dangerous systems that design pandemic viruses without a shred of emotion, and likewise can imagine sublimely emotional and empathetic chatbots unable to either cause much harm or solve any real problems for us either. So I prefer the AGI definition I expressed largely because it avoids those murky consciousness questions and focuses on ability to impact the world in measurable ways.
To continue your engine analogy, I think we can definitely agree that the “check engine” light is firmly on at this point. I think that the drawing a line in the sand for AGI vs. “very powerful LLM” is, at best, subjective, and distracts from the fact that the LLMs/AIs that exist today are already well capable of causing the widescale damage that you warn of; the technology is already here, we are just waiting on the implementation. Perhaps what I mean is that we have, in my view, already crossed the line—the timing belt has snapped, the engine is dead, but we’re still coasting on the back of our existing momentum (maybe I’m over-stretching this analogy now...).
We may have an object-level disagreement here. I agree that the “check engine” light is one, and that current AI can already cause many problems. But I also expect that there is a qualitative difference (again, not a bright line, though) between risk from today’s LLMs and from AGI. For example, current AI evals/metrology have established to my satisfaction that the risk of GPT-5 class models designing an extinction level virus from scratch is extremely low.
That’s a fair point, but if we aren’t arguing about “consciousness,” and we have grounded our definition of “AGI” in, essentially, its capacity to do damage, then I think these kinds of tests fall into the same category as GDP in economics: a reasonable corollary but ultimately unsuitable as a true metric (and almost certainly misleading and ripe for abuse if taken out of context).
Absolutely, valid concerns. Folks in AI evals/metrology are working very hard to make sure we’re measuring the right things, and to educate people about the limitations of those metrics.
For sure! I just don’t feel the need to wait for this technology to be relabeled as “AGI” before we do something about it. If your concern is their ability to act, as the agents on Moltbook act, (let’s say) “semi-spontaneously,” then we are clearly already there: all we are waiting for is for a person to hand over the launch codes to an agent (or put a crowd of them in charge of a social-media psy-op, prior to a key election, etc.).
Yes, I am not suggesting that we wait. We should be acting aggressively now to mitigate risks.
You say that AIs would need to be “qualitatively” different to current generation models to do pose enough of a threat to be worthy of the “AGI” label. Please could you outline what these qualitative differences might be? I can only think of quantitative differences (e.g. more agents, more data-centers, more compute, more power, wider-scale application/deployment, more trust, more training data—all of these are simply scaling-up what already is and require no truly novel technology, though they would all increase the risk posed by AIs to our society).
The qualitative differences I’m referring to often involve threshold effects, where capabilities above the threshold trigger different dynamics. Sort of like how the behavior of a 51 kg sphere of enriched uranium is a very poor guide to the behavior of a 52 kg sphere at critical mass. Some concrete examples include virus design (synthesizing a high-lethality virus with
$R_{0}>1$
is a pandemic, and lower than that generally isn’t), geoengineering (designing systems capable of triggering climatic chain reactions, such as superefficient carbon-capture algae), nanotechnology (designing nanobots that can self-replicate from materials common in the biosphere). In all those cases, the dynamics of a disaster would be wildly different from an AI malfunction at lower levels of capability.
As for your point that you, personally, are concentrating on the individual response within the wider community of alarmists who, collectively, are concentrating on both the collective and the individual response: thank you for clarifying this, it is important context. I definitely agree that both avenues need exploration and it is no bad thing to concentrate your efforts. I would say that, for my rope, the collective response is where I think the overall course will be set, but when collectivism fails, then individualism (or, more realistically, smaller scale collectivism) is the backstop. In this vein, I think that point 10 from your original article is the absolute key: it won’t be your basement full of tinned food that saves you from the apocalypse: it will be your neighbours.
Perhaps I worded this in an unclear way. I am personally concentrating mostly on the collective response. But this particular post is about the individual response, partly because there is less clear and accessible material about that than on the collective response, which is a major focus of many other LessWrong posts.
Many thanks for the thoughtful exchange!

John-Clark Levin 12 Feb 2026 3:58 UTC
2 points
1
in reply to: xle’s comment on: Eleven Practical Ways to Prepare for AGI
Very helpful amplifications, xle! Much appreciated.
I really do get the appeal of the “spontaneously express … complex desire or emotion” framing, but if I’m understanding you correctly, the whole thing basically hinges “spontaneous”, since AI can already express complex desires and emotions when we prompt it to. But agents on Moltbook are already expressing what purport to be complex desires and emotions even without any prompting. If this doesn’t count because the agents were first instructed to go do things spontaneously, we start to see that “spontaneous” is a very slippery thing to define. Ultimately, any action of an AI we create can be traced back to us, so is in some sense not spontaneous. So it’s worth thinking as concretely as you can about how you’d define spontaneity clearly enough that it could be proven by a future scientific experiment, and in a way that would resist post hoc goalpost-moving by skeptics.
Your “90th percentile” operationalization is a good way of getting at roughly the AGI definition I’m endorsing. One issue to flag, though. AGI will have massive impacts, and it will be important to have some warning. If the minimal thing that would increase your credence of “AGI soonish” is AGI itself, you’d be committing yourself to not having any warning. Yes, the engine sputtering and dying is a very solid signal that you’re out of gas, but also a very costly and dangerous signal. So there’s value in figuring out your equivalent of a fuel gauge warning that lights up while the engine is still running fine—something pre-AGI that would convince you that AGI is probably coming soon.
What I’m getting at about mathematics is just that it’s a domain that’s effectively independent of human culture, so not subject to anthropomorphization in the way that writing haikus or saying “I love you” is.
I agree that who holds the proverbial launch codes is of extraordinary importance, and that we must marshal enormous civilization-level effort toward governing AGI responsibly, justly, and safely. That is, in fact, a much more central concern of my research than the subject of this post, which is individual-level preparedness. We absolutely need both. But I am making the additional claim that AGI will have the capacity to act with meaningful agency—to decide on targets and launch itself, in the nuclear weapons analogy—and that this introduces a qualitatively different set of challenges above and beyond the political ones. I don’t intend it as an absolute line in the sand between AGI and today’s LLMs, but I do claim that qualitative difference to be very important.
It’s good to see on how much we’ve come to agree on here, despite approaching this with different framings.

John-Clark Levin 11 Feb 2026 2:55 UTC
2 points
1
in reply to: xle’s comment on: Eleven Practical Ways to Prepare for AGI
Thanks — glad you found that helpful! That’s a good clarification. One thing I invite you to consider: what is the least impressive thing that AI would need to significantly increase your credence in AGI soonish?
To clarify, the definition of AGI I’m using (AI at least at the level of educated humans across all empirically measurable cognitive tasks) does not entail any claims about true consciousness. It’s narrowly a question about functional performance.
I think AI progress in very pure fields like mathematics is our best evidence that this isn’t an anthropomorphic illusion—that AI is actually doing roughly the same information-theoretic thing that our brains are doing.
Your outside-view skepticism of doom scenarios is certainly warranted. My counterargument is: should a rational person have dismissed risks of nuclear annihilation for the same reason? I claim no, because the concrete inside-view reasons for considering doom plausible (e.g. modeling of warhead yields) were strong enough to outweigh an appropriate amount of skepticism. Likewise, I think the confluence of theoretical reasons (e.g. instrumental convergence) and empirical evidence (e.g. alignment faking results) are strong enough to warrant at the very least some significant credence in risks of doom.

John-Clark Levin 10 Feb 2026 19:18 UTC
1 point
0
in reply to: cubefox’s comment on: Eleven Practical Ways to Prepare for AGI
I agree about overall deflation, and relative exceptions for land/housing barring policy interventions.