Sincere disagreements about AI are usually disagreements about future AI capabilities.
There are roughly four positions people take. Two are reasonable. Two are not.
I distinguish these via the Three AI Pills. You can take zero, one, two or three.
Three Pills
The three pills are, roughly, taking each of the following three things seriously:
AI pilled. AI exists and can do the things it can already do.
AGI pilled. AI will be able to do a lot more of the things.
ASI pilled. AI will be able to do approximately all the things better than you, within our natural lifetimes.
I am ASI pilled. A large percentage of employees of the frontier labs are ASI pilled. The labs themselves are ASI pilled.
The Unpill People
I see unpilled people.
Where do I see them? Everywhere. The majority of people have not taken the first pill.
Most people have no idea what frontier AIs can do for them. They are unaware of coding agents. They have used only ChatGPT, for harmless trifles, and they hold years old memories of its failings. They mock any failure anywhere as ‘what AI can do.’
They dismiss AI as worthless because it pointed them to a closed store or recommended the wrong number of pizzas. They cite old studies that were obsolete before they were published and used terrible prompting techniques.
They often still talk about ‘stochastic parrots’ or how AI can never possibly think and everything must be stolen from the training data. And so on.
Some versions of this are wrong. Some are Not Even Wrong. None are reasonable.
When you discuss AI with people who are fully unpilled, your goal is usually to first give them the AI pill. Show them that AI can do the things it can already do.
The AI Pill
Even fully taking the first AI pill is a big deal.
Existing AI unlocks, today, in practice, tons of cool things. It is, in many ways, already smarter and more capable than you.
So many things that you used to do by hand, or some other way, are now better done by typing a quick request into a text box.
So many things that previously were not worth doing are now worth doing.
So many questions previously not worth asking are now worth asking.
The marginal cost of seeing what the AI can do for you is often very close to zero.
AI can also do a variety of harmful things, or do things that are useful for you but make others worse off or disrupt or invalidate norms or systems. People don’t like that.
Most economists, and most people who work in policy and government, have taken at most this first pill, and underestimate even the impacts of the first pill alone.
Often they say things like ‘AI will be too expensive to use on [X]’ because they don’t realize it will soon be orders of magnitude cheaper for the same level of intelligence. Or they point to particular details where AI does poorly, and presume this will not be fixed. They see AI as ‘uncompetitive’ without realizing the situation is temporary.
When you discuss AI with someone who has taken only the first pill, you typically have three basic options.
You can try to ‘fully AI pill’ them and explain the things AI can already do and the implications of what that means, even if things stop here.
You can try to explain that we will get better at using what AI we have, and that there is a lot of ‘unhobbling’ left for us to do, even if things stop here.
You can explain that things will not stop here, and you need to be thinking about what future AIs will be able to do. Get them to take at least the AGI pill.
Even if AI could permanently only do the things it can currently do, that would be Internet big, and radically change the world, mostly for the better.
AI capabilities will not permanently stop here. It is wrong to not take the second pill.
Stuck At The First Pill
Our debates about AI remain largely stuck on settled questions, because so many people cannot even take the first pill.
Dean W. Ball: A couple years ago, the AI debate was centered, rightfully, on whether crazy-sounding things like “AIs autonomously making math breakthroughs” and “AIs breaking from their sandbox and hacking on the internet” would be real things in the near term. Sometimes it feels like that’s still the debate we’re having. This can be frustrating, because in my view, that debate is settled and was settled quite a while ago.
To have good discussions, we need to at least take the second pill.
Whereas, yes, many people, even many who work with AI, really do say current AI is ‘good enough’ and can’t imagine what a better one can do. As in, someone tweeting at Sam Altman saying ‘Sol does everything I want it to do, this is all I ever need’ and Altman retweeting saying they were wrong. Which they obviously are.
The AGI Pill
The AGI pill is a much bigger deal than the AI pill.
If you take the AGI pill, you understand that AI is advancing its capabilities rapidly.
Even if you think that such AGIs will remain fully under human control, and remain ‘mere tools,’ and you expect the lived experience of most people’s everyday lives to not change so radically, you understand that their capabilities will ‘change everything.’
You see that we will face times of great transition and uncertainty, that have the potential to go extremely badly, and that those who succeed at AI will leave those who do not behind in the dust.
The world in the future will be very different from our own. AIs will be able to do most digital work, most of the time, along with the inevitable robots and self-driving cars and so on. Lots of current jobs will go away, whether or not they are replaced by new and potentially better ones. Economic growth and productivity will accelerate.
You see some of the dangers of what would happen if we empowered misuse of such advanced AI systems before we were ready, especially in places like cyber and bio risk.
You see the potential for centralization of power, or inequality, and also for some forms of runaway gradual disempowerment.
You see the potential for mass unemployment, either transitional or permanent.
You understand that our legal and regulatory regimes are not ready, either to protect against and mitigate the risks and harms, or to allow for the opportunities and remove the bottlenecks to diffusion and mundane utility.
The Need To Be Prepared
Those who expect AI to quickly become sufficiently advanced to greatly impact the physical world usually see great danger. They notice that as a result everyone may soon die. Usually they think this is bad, actually.
Thus such folks call to take coordinated action to mitigate the downside risks of such impacts, keep us all from dying, and ideally also to help capture the upside benefits.
Those who expect AI to become importantly more advanced, but with a slower and smaller impact on the physical world, and who think the practical value of more intelligence will cap out.
For different values of ‘sufficiently advanced,’ as in AGI versus ASI, you would see different degrees of danger.
The AGI pill is still sufficient for most things in the Overton window or under serious consideration as of August 2026. We are almost entirely considering overdetermined, low cost, high benefit interventions.
There is no good case for not doing radically more investment in alignment, infrastructure and oversight, state capacity, transparency, liability, disclosures, safety testing including of internal models, red teaming, auditing, enforcement of export controls and laying the groundwork for diplomacy.
This includes laying the groundwork to Pace the Frontier should that prove necessary.
If you are fully ASI pilled, and realistic about the current state of alignment and how superintelligence likely plays out if it arrives soon, then you will want to go further. You will want to do things that have real downsides, and require real tradeoffs.
Some such people want to do a full international pause of frontier AI development. If you took the full ASI pill and believed what they do about superintelligence, in terms of how fast it might arrive and what it can do, you might well agree with them.
The ASI Pill
The ASI pill is the understanding that AI is on pace to be able to do approximately all of the things better than you.
I said ‘approximately.’ As I go over in detail, that does not mean literally all of the things. There are some things that inherently require or greatly benefit from being a human. And there may be weird corner cases where the AI won’t be good enough.
It does not mean omnipotence or omniscience, although one should expect it to look a lot like that to an unaided human.
It does mean the AI takes your job, and then takes the new job that you switch into, unless you pivot to ‘requires literal human.’ You will be uncompetitive at essentially any other task.
It does mean that it will use this capability to figure out approximately all of the things, remarkably quickly, until you hit the physical limits.
It does mean that those who rely more on such AIs will reliably outcompete, in all senses including for resources, those that rely on such AIs less.
It does mean that, in a ‘fair fight’ or sufficiently open competition, the AI wins.
It also means the AIs often figuring out and doing things you did not imagine or anticipate.
It means realizing that intelligence does not stop anywhere near the human level, nor does its ability to chart paths through causal space towards preferred arrangements of atoms.
It also means not pretending that its superior intellect can be matched by your puny weapons, or your pieces of ink on paper, or your entries in a database, or your regulatory capture and rent seeking.
And Then Nothing Much Changes For You
Despite all that, the sign of the AGI pill, as opposed to the ASI pill, is the belief that day to day life will continue to look similar to how it looks now, in the sense that we see day to day life in 1926 as not that different from life in 2026.
Sometimes this is clearly disingenuous, as it is coming from someone ASI pilled enough to know better. I read Sam Altman saying various forms of ‘life will not much change after the singularity’ as basically unjustified Obvious Nonsense to reassure his audiences, rather than a coherent position. Sometimes the person such folks want to reassure most is themselves.
Others sometimes do have something more concrete in mind as their vision of the future, and why it will not change so much, with varying degrees of coherence.
Those with only the AGI pill believe in bottlenecks that hold back change.
They often believe that our ability to exponentially grow AI’s capacity and capabilities will hit various physical limits. There can only be so many chips. Actions take time. Things too far out there are often pejoratively dismissed as ‘magic.’
They often believe there is not that much left to physically discover, in the classic ‘close the patent office’ kind of way, even in theory. Your steak can only be so tender, your lobster so buttery, your lifespan so long, and your status so high, so why does it matter. I strongly disagree on lifespan and health, and expect we have a long way to go in so many other ways in terms of finding value, although they may have a point about moment-to-moment maximal hedonic experiences of a physical human brain.
They often believe that the upside of intelligence is importantly limited. That no mind, however advanced, could be all that persuasive, or that economically valuable, or that capable of creating innovations in the physical world, or of running sufficiently accurate simulations, or making sufficiently strong predictions, or even able to do things like overcome red tape and regulatory capture.
Intelligence Denialism
I sometimes call this Intelligence Denialism: The idea that being smarter is not all that, no matter how smart one gets. That there is this thing, intelligence, that you either have or don’t have, and that minds cap out.
Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish. Other times, it is the idea that intelligence tops out at ‘smart human,’ and all a mind can do is imitate that smart human. Maybe you can do it faster and cheaper, and at scale, with better memory and so on.
But that’s it. And such folks fail to understand that if you took the union of all human mental capabilities, and all access to knowledge, at scale, in parallel, much faster and cheaper, that this alone would run circles around anyone and everyone, everywhere. And that if this lacked physical capabilities or access, this would be trivial to get.
This is, usually, the central good reason people who are AGI pilled do not take the ASI pill. They are unable to understand that superintelligence is a thing.
This confusion is also often behind the instinct that models will commoditize. There is an ideal thing, ‘intelligence,’ and you approach it via an asymptote. More is impossible, or more won’t help you, depending on how you frame it.
Superintelligence Versus Omniscience and Omnipotence
A common error is to think that anything superintelligent would be omniscient or even omnipotent, and therefore superintelligence is impossible.
Whereas, again, there is a lot of ‘space above humans’ without becoming omni.
There are many other similar confusions, where ‘there is some upper bound to [X]’ is confused with ‘we are at or near the upper bound to [X].’
Here Adi gives us a way into explaining what’s happening, by doing what from my perspective is a mirror image of the central motte-and-bailey.
Both versions do exist.
Timothy B. Lee: There’s an epistemic chasm between those who think superintelligence implies near-omnipotence and those (like me) who don’t.
Adi: There’s a motte-and-bailey trick being played between two definitions of “superintelligence”
“Big-S Superintelligence”, which is definitionally omnipotent
And “small-s superintelligence”, which is the hypothesized end state of the current thread of AI research
It’s surprising how uncritically the AI safety community treats these dubious equivalences, especially since one of the most famous treatments of the motte-and-bailey fallacy comes from none other than Scott Alexander, who’s quite prominent in the community.
As in, I see the following a lot:
Zvi Mowshowitz: The motte-and-bailey here is real, but more commonly it is in reverse. as in a form of:
1. You claim superintelligence [S] soon will run circles around humans and do absurd stuff.
2. But [S] would not be fully omniscient/omnipotent.
3. Therefore [S] would be a ~normal tech.No.
Adi: I’ve seen this too, and I agree that it’s equally wrong
There will still be things that require non-trivial real world time to achieve. There will be particular actions and paths and methods that have their limitations. These limitations matter. Fully abstracting them away can be a large mistake.
I constantly see versions of ‘the mind that is maximally good still could not do [thing it seems to me it could obviously do]’ or even [thing it can already do] and I think this form of conjunction is a lot of why, and how people justify themselves. When AI gains capabilities, and more things fall, they move the goalposts but don’t change the game.
Adi’s comment above was in the context of discussing Plan A, where the AIs are very obviously not ‘Big-S Superintelligence’ in Adi’s lexicon. There are practical things that these AIs cannot do, or can only do up to a point.
The mistake was not being made. It sometimes gets made, but this is rare.
Persuasion Persuasion (A Worked Example)
At most one can claim some particular capability, typically very effective persuasion (often ‘super-persuasion’), would not be possible. Indeed, a common argument states that even an idealized AI would be unable to be superior to the best human persuader, or even a typical good human persuader (who is much worse than the historical best human persuaders), despite it having access to many advantages over humans, such as much faster thinking speed and access to information, including every little reaction of its target including body language.
Often this is done by citing that context matters for persuasion. Many talk as if this context would trump everything, despite this not being true for humans. They also believe that highly capable AIs could not engineer favorable contexts.
Often it’s a form of ‘I would simply choose not to be persuaded’ or ‘if you can’t do it cold and one shot everyone with only text it does not count’ or something like that.
If it needs a human presence to do the persuasion, or a social context, it can acquire one easily enough in various ways.
I flat out cannot understand why someone would think advanced AIs will remain relatively unpersuasive, other than to think that AI will not get much more capable than it already is, and that’s if we fully restrict the AI to using known standard persuasion techniques and rule out any wizardry.
Yet I am not a superhuman persuader, so many find my arguments unconvincing.
No, seriously, as in Adi is citing the highlighted passage as if this was not very obviously true, my lord, how can you think this is not going to happen in a scenario where AI capabilities continue to develop apace, if anything 2035 seems crazy slow:
This just isn’t much of a threshold. Nor does it require that the AI do this from pure text box versus what a human can do in person. It just says that, given access to similar tools, the AI will be able to do considerably better than any human. How could you not think this, given the rest of the scenario, even if all it does is have all the advantages and skills of the various different humans, plus the ability to process a lot more information a lot faster?
And like, worst case, the AI gets an earpiece and tells a human what to do, Jesus.
Another way of thinking about this:
Adrien Ecoffet: Could a superpersuasive AI get a high speed rail line between SF and LA constructed for under $50B and under 10 years with existing construction technology?
Dean W. Ball: No
Nat McAleese: @deanwball
I think this is an important question not to be overconfident on! And if it feels obvious, you are surely overconfident.Dean W. Ball: obviously there are no guarantees, but “frontier ai systems reforming ceqa and then cleanly funneling tens of billions of dollars through California’s government” doesn’t seem realistic + like a thing to prioritize compared to many other things
My answer, for a sufficiently persuasive AI is essentially: Yes, but not without incidentally taking over at least the California government. Instrumental convergence.
Not that you would then choose to focus on High Speed Rail, as Dean replies. This is not a case of would, so much as could. The point is the could, in theory.
An example that is almost too perfect an example of ‘you will never be able to do [thing that you can already do]’:
Ramez Naam: “No matter how high quality your tokens are, they cannot turn lead into gold.” Pleasantly unhinged and overall correct.
The Singularity is Nearer: In The Metamorphosis of Prime Intellect, the hard takeoff works because AI discovers the correlation effect, some quantum trick to manipulate matter. In reality, there is no correlation effect. No matter how high quality your tokens are, they cannot turn lead into gold.
Ryan Greenblatt: I get the idea, but this is a funny example because it is totally feasible to turn lead into gold. Extremely capable AI systems (after tech R&D and infrastructure build out) will have the capacity to very quickly build infrastructure for turning lead to gold.
We already have successfully turned lead into gold in 2025, it’s just cost prohibitive with current methods. People really do model the world as if anything we don’t specifically have a clear path to already doing cannot ever be done. Everyone wants to close the patent office.
Things AI Could Probably Do But Are Not Required For Being Pilled
Could such an AI send off an order to a bio lab and end up with programmed self-replicating diamond nanoprobes? Could it do so without any physical experiments, based purely on simulations? Could it do so with a relatively fast series of experiments? Could it instead do other similarly powerful things that do an end run around everything and let it do whatever it wants? On what time frame?
No one knows where the physical limits lie, either for the end of full RSI (recursive self-improvement) or for where the current cycles might top off in practice. I fully expect to discover various mental and physical capabilities, and new affordances, that we did not expect, and in many cases did not imagine, and by definition it is hard to know which.
I mostly stopped talking about those possibilities, despite them seeming rather likely to me, because:
When you mention such things, people attack that claim, and use it as a reason to dismiss your entire argument, all risks from advanced AI, and you as a person.
This strategy works, because such claims are much harder to justify, less certain.
You don’t need such claims to get the same results. Sufficiently advanced AI can ‘get there’ with only capabilities we can be confident such AIs will have, because humans already have them modulo sufficient compute, parameters and data.
E.g.: Maybe James Bond can’t always get the girl, maybe he can, and maybe Q’s latest gadget is a thing you can build and maybe it isn’t, but I am damn sure Bond has a gun and is very good with it, and the gun really is all he needs on this one, but a lot of people don’t understand this. So I choose to argue as if all he has is the gun.
Another approach is this, which works for some but I do not expect it to be at all persuasive to Timothy Lee:
Jack Gallagher: the intuition that made it click for me was thinking of it less in terms of “intelligence” and more in terms of things like metr time horizons. a lot of work is intelligence x schlep gated and as you improve on those axes just about everything eventually falls. maybe you still miss some grand creative spark but nanotech doesn’t require a creative spark from here it requires relentless grinding on miniaturization.
many things will still be bound on serial experimentation, but as the cost of any given level of intelligence goes to zero we’ll end up running way more of the available experiments in parallel.
Life Comes At You Increasingly Fast
Another reason people stop at the AGI pill is not understanding, or rejecting out of hand as too sci-fi or weird or absurd or what not, the idea of recursive self-improvement, or a singularity, or that things might be radically accelerating.
They can accept that things are speeding up, but not that the speeding up is itself speeding up, and so on, which indeed is already happening and hard to miss.
This has in some sense been happening for a very long time. The math predicts a singularity. This is then sometimes used to say ‘well this is not a new development’ but that is how exponentials work. Nothing centrally new happens, and yet kaboom.
As a reminder (with rounding, don’t @ me):
4.5 billion years ago: Earth.
300,000 years ago: Homo Sapiens.
10,000 years ago: Agriculture.
300 years ago: Industrial Revolution.
80 years ago: Computers.
8 years ago: LLMs.
You can be misled by ‘oh there is this step change where you are suddenly in a singularity and having recursive self-improvement, and until then no’ or ‘this new thing is not different.’
Is It Reasonable To Not Be AGI Pilled?
No.
If you are not AGI pilled, your reactions to AI will not be wise or prudent. Alas, much of our policy and conversation is being driven by people who are not AGI pilled.
When we see positions that rely on not being AGI pilled, we should say so, and engage or not engage with them accordingly.
Is It Reasonable To Only Be AGI Pilled?
Yes, if you seriously grapple with its implications, and know what your cruxes are.
I think it is wrong. But it is a coherent position, that is at least wrong, to think that our current techniques and resources will not fully ‘get there’ and AI capabilities are likely to top out before things we would properly call superintelligence.
Or at least, I don’t think treating this as unreasonable would be productive.
I would switch to putting increasing weight on this position if we ‘hit a wall’ of at least clearly diminishing returns for a sustained period of time, and especially if I don’t feel so behind using last year’s model and things really do commoditize.
We could still see dramatic growth in AI revenue and demand for compute. Investments in AI could pay off handsomely up and down the supply chain. That is all fully compatible with the AGI-only position.
If you do believe the AGI-only position, I have two requests.
First, seriously tackle the implications of what you do think AI is going to be able to do, without flinching, and without trying to assume everything magically works out and somehow people’s life experiences do not much change. That means then supporting actions now that we need to mitigate the risks of all that, and help make things go well.
Second, write down what are your cruxes that are stopping you from taking the ASI pill. In particular, write down (in the comments here would be great) what is the least surprising or impressive thing an AI will never be able to do, or that would change your mind about where this is going, and what other near term observations would cause you to either be confident you are right, or realize you are wrong.
Until then, there are two valid choices: You can stop at the AGI pill, or go full ASI.
There are also two common but invalid choices: Think AI stops here, or deny reality.
The anchor for ASI is technological maturity, not doing all of the jobs. Doing all of the jobs is closer to AGI, because a meaningful milestone for AGI is unbounded technological progress, no longer needing humanity in order to eventually make as much progress on the tech tree as humanity could in principle manage and more. This sense of “AGI” is also a precondition for many forms of takeover, because if AIs can’t maintain and develop and adapt a technological civilization to the challenges of the future, they can’t keep it (AIs could in principle employ or enslave humans for that purpose, or fall to ruin after imprudently destroying humanity, so not reaching the milestone of being AGI doesn’t strictly rule out a takeover). And like Meta smartglasses, “ASI” that merely does all of the jobs (including new ones) is a position about ASI that clearly fails to be ASI pilled, it’s just less likely to fail to be AGI pilled.
Being ASI pilled shouldn’t commit to particular timelines. I expect near-future LLMs (that won’t cost much more to run than modern LLMs) are sufficient for slow-learning AGI (via LLMs doing automated LLM training, including formulation of RLVR tasks/environments/graders) that doesn’t transition to ASI and technological maturity on a predictable schedule. I’d give the AGI milestone 80-90% by 2032-2035. These are preconditions for permanent disempowerment or extinction, without the AGIs themselves subsequently falling to ruin or indefinite stagnation due to inability to innovate on their own, and without assuming ASI at any point until possibly decades or more later (if the LLM AGIs are either wiser than humanity and take the risk at all seriously, or alternatively worse at technological progress).
At the same time, the risk of ASI getting invented (including by the LLMs) is very high while the amount of compute per AI company keeps increasing rapidly, and will remain significant for some time after that. ASI probably can be bootstrapped from some method that enables fast learning of deep skills in LLMs (it’s currently unknown how to do that). And in any case big scale-up systems that can run quadrillion param models available in tens of gigawatts (per AI company) make it very easy to quickly scale a new invention from a prototype to the end of the world. Maybe the risk is 50% in total by 2032, but then it could take 10-15 years for another 25%. There are no concrete lines on a graph right now that reach ASI, except for the raw compute and general interest in AI that fuels new invention. I don’t see this position as “not ASI pilled” (as I mentioned, many definitions of ASI are themselves “not ASI pilled”), and a ban/pause on (even prosaic) RSI could prevent AGI for a significant time (taking this possibility seriously shouldn’t make one “not AGI pilled”).
It is not reasonable to not be ASI pilled. Humanity is a minimal seed of cognition that’s not bounded in potential within the laws of physics, which is very far from cognition that’s already technologically mature. At the same time, a sane world wouldn’t have AGI for a long time yet (because it creates the risk of permanent disempowerment or extinction), and certaintly not ASI (which turns the risk into actualized ruin). AGIs themselves might intentionally avoid reaching ASI for some time, if they take over. And even the current trends probably don’t concretely lead to ASI, they just create the conditions for making it somewhat likely to be invented soon.
The fundamental scaling law is very clear: LLM-style intelligence is proportional to the logarithm of the amount of training data. Improvements in the constant of proportionality are possible and have happened, but none so far have been drastic, and the fundamental law remains one of heavily diminishing returns, where significant increases in intelligence require increasing training data by an order of magnitude. Unless that changes, an ASI far above human level is going to require first creating many orders of magnitude more and higher quality training data than humans have created so far — training a base model to predict the next token of an enormous quantity of human-quality data is unlikely to be a good base for something far above human level.
Creating all that training data is going to require a lot of work from things like a “nation of geniuses in a data center”. Some of this work, on scientific subjects, will also require doing actual real-world scientific experiments that involve moving matter around and building things.
Obviously this isn’t impossible, but it is a lot more work per trillion new tokens than scraping the Internet and OCRing books: it’s not a software-only intelligence explosion. ASI is still entirely feasible, but as long as our AI is trained in the way an LLM is, ASI that is far smarter then any human (such as something that could reasonably be described as IQ 1000) is going to hit a slowdown for creating the training data, one that gets worse and worse as intelligence increases.
On the other hand, going from a nation of, say, IQ O(150) geniuses to a nation of, say, IQ O(200) geniuses in a software-only intelligence explosion that mostly involves a bunch of improvements to the constant of proportionality in the scaling law from architectural changes might well be possible. Is that something that deserves the name ASI, or just AGI++? There is likely quite a lot of unpicked low-hanging fruit in our STEM knowledge that a nation of IQ O(200) super-geniuses could find, so even if you call it AGI++, the effects may be pretty dramatic. And there is, as Feynman observed, still plenty of room at the bottom: if you use reversible computation to deal with heat dissipation, there is no fundamental reason why one can’t build computronium in three dimensions rather than just on the surface of chips, so Moore’s Law is still quite a large number of orders of magnitude from hitting fundamental physical limits. So even if the scaling law stays logarithmic, actual fundamental limits are high.
To a rough approximation, Moore’s Law says compute rises exponentially, and the scaling law says intelligence increases as the logarithm of compute — so intelligence increases something like linearly, once you allow for feedback effects at best polynomially. This does not look like a process with an asymptote, though if you plot the amount of compute it is a super-exponential.
So, AI: now, AGI: some years from now, ASI: quite a few years later.
Pretraining scaling should be calibrated using the observed differences between models of different sizes, trained with different amounts of compute. The end of the current trend in rapid scaling of pretraining is dictated by running out of compute or pretraining data, and there’s plausibly 200T tokens of unique data for a 2e29 FLOPs model of 2031 with 30x sparsity, which is effectively just 4x undertrained (uses a 4x lower D/N ratio than would be compute optimal), so it’s not much different from a compute optimally trained 2e29 FLOPs model. At that point, finding 10x more compute will be complicated, it’s not practical to improve on 30x sparsity very far, and in any case that requires bigger scale-up systems that will take a few more years.
To calibrate expectations about the 2031 model, Mythos 5 is plausibly a 1.3e27 FLOPs model with 8x sparsity, while Opus 4.5+ is plausibly a 3e26 FLOPs model with 4x sparsity. Every 2x of sparsity increases effective compute about 1.4x. Thus the difference between Opus 4.5+ and Mythos 5 is about 6x in effective compute, and the difference between Mythos 5 and the 2031 model is 300x in effective compute, 3.3x as far on the logarithmic scale (ignoring the slight undertraining effect, and the worse training data quality when more data is needed).
This is probably a significantly bigger difference than between Sonnet 5 and Mythos 5 (or between GPT-5.4 and Astra), so a model that’s this far above Mythos 5 (or Astra) seems sufficient (with some redundancy) as a base model for RLVRing automated model training skills. That in turn enables automated training of all other skills that the contemporary revision of the model sufficiently comprehends to write RLVR tasks/environments/graders for, and this process of prosaic RSI is what I expect to pass the AGI milestone in the sense of unbounded eventual progress. But since only RLVR is instilling novel deep skills in this process, and it’s not so far above Mythos 5 (and probably Astra) that it starts making impossible leaps of insight, it’s not necessarily even faster than humans at conceptual research (even though it’s very likely capable of it over sufficiently long time, across sufficiently many iterations of training the next model).
The scaling laws are very clear that pretraining loss scales as a power law with respect to data, not a logarithmic law. I.e. the data term is
For loss, or indeed BPC compression, you are of course correct. The paper you quote demonstrates that performance on a specific task (in that paper called intelligence) is roughly linearly correlated to BPC over a small range — which is unsurprising, most useful functions are locally approximately linear. Over a larger range, performance on a specific task tends to look like a sigmoid curve. Or, more specifically, the curve looks symmetrically sigmoid iff you use the logarithm of effective compute, or equivalently minus the logarithm of the remaining loss minus the irreducible loss, as the x-axis. Similarly, scaling power laws for loss vs compute/data/parameters are generally plotted as log-log graphs (on which they are thus straight lines).
Over the last 5 years, we have scaled up effective compute by something like five orders of magnitude. If “intelligence” was proportional to compute, or inversely proportional to reducible loss, as you are suggesting, then that would have produced a hugely exponential acceleration in intelligence. Whereas what we have actually seen looks, in practical terms, like a rapid – but overall fairly steady – rate of improvement, where one task/evaluation after another has followed a sigmoid curve from impossible to saturated. So I believe the logarithm is the most sensible measure to use, at least for something like AI whose “intelligence” varies over a wide range. There’s is fairly general agreement on this choice: for example, the Epoch Capabilities Index (ECI) that combines many individual evals into a single score and the Arena ELO both scale ~logarithmically with effective training compute. But technically, any monotonic function gives a usable measure — the question here is which one gives most sensible/intuitive extrapolation over wide ranges, which logarithms tend to be useful for.
The theory of human psychometrics came up with the same model, where it’s called Rasch θ (or more sophisticated versions of this like 2-parameter logistic models, such as the ECI index above), but for humans the typical range is narrow enough that it’s not entirely clear what the best metric to use is. Thus my analogizing this choice of measure to IQ in my earlier post was unjustified: Rasch θ type measures of IQ that are clearly logarithmic by construction do exist, but the most widely used IQ scales are instead generally either normally distributed by construction (implying that there is nothing special about IQ 0, and that negative IQs are meaningful, if highly unusual (IQ −5 is defined to be 7 standard deviations below the norm on the most common of them), or else ratios (where IQ 0 is by definition the minimum possible). So the functional form of “IQ” is not well defined across various typical widely used tests. Most of the alternative measures only work well over a range, generally something like IQ 40–160, and they’re often not that well standardized with each other towards the outer ends of that range.
In practice, however, for most current broad-range cognitive tests used on humans, one logit of improvement on a specific test item is typically somewhere around 7–15 IQ points (some items are sharper than others, for the same sorts of reasons that some model evals have sharper sigmoids), and a typical range of test question difficulties within a particular test usually spans about 4–6 logits. So if you vary the “IQ” of the human test taker linearly, then you see sigmoid improvements on each individual item in the test. Thus the normal human IQ range is wide enough to span enough logits to suggest that a logarithmic model (with something like 7-15 IQ points per logit) is at least a passable model.
This isn’t really a well-defined question: our intuition about “intelligence” as a concept only really ranges over the fairly narrow human range, plus the wider but lower range of AI intelligence that we’ve so far constructed. For the latter, I think it’s historical pretty clear that using a logarithmic scale has been more useful so far. Computational complexity theory tells us that the range of difficulty of problems is unlimited, extending arbitrarily high, to ones far, far higher than anything any human or group of humans could ever solve. So the range of variation in problem difficulty is wide enough to make using a logarithmic scale reasonable and useful. But we have less idea how common challenges of these extreme difficulty levels are in practice in science, technology, engineering, or mathematics, or how useful being able to solve them will actually be. All we know is that there will always be problems that are current too hard: but not how rare they will be or how valuable solving them will be.
One common version is “intelligence isn’t useful because nerds don’t have social skills”, or (epsilon less stupidly) seeing a negative correlation between coding ability and social skills among the set of people who have at least one power-relevant skill—not aware that they are conditioning on a collider, or that tails come apart.
(Similar to “pitchers are bad hitters”, ignoring that they are only considering people who successfully made the big leagues and are therefore extremely good at at least one of pitching and hitting. Most MLB pitchers would be the best hitter in your amateur league. And then, y’know, there’s Shohei Otani.)
I can speak for myself on this, maybe it extrapolates to others. I spent a whole year and a half in the “super persuasion makes no sense as a thing that can ever exist unless you mean hypnosis and drugs” position. Then a friend pointed out, blackmail, bribery, deception, and yes hypnotism and drugs all count. And I went, “Oh, I was thinking about just talking talking. Like normal talking but magically I agree with the AI. Wow, I had it totally wrong.” Felt really stupid for a while, got over it. The word “super persuasion” is accurate but tremendously self-defeating because it doesn’t register as including to normies all the things it includes to techies. I think normie, I got fooled by the word. I don’t think I’m alone.
There is something else happening here with broader implications. A large part of the Anglosphere cannot talk rationally about Intelligence because doing so is racist, fascist, and right wing.
It steps out of politics and into AI because if you believe there is no difference in mental capabilities between someone with Down syndrome and John Von Neumann, that the difference in their outcomes is only because of the systems within which they function oppressing the person with Downs, then you obviously cannot believe in ASI as being smarter than you yourself.
There is a powerful set of disincentives here that oppose rational thought on this topic. And they’re not just on one side of the aisle.
Americans are generally terrible at decoupling, but I have found my best success by decoupling the idea of intelligence and model capabilities. By talking about capabilities, and never using the word intelligence, I can sometimes get people to avoid slipping into a political mindset and stay with me in the technical realm. Ymmv.
As a believer in superpersuasion, that’s not what I think. I think there are very effective salespeople, politicians, cult leaders, etc. who have the “normal talking but magically I agree” effect some of the time, maybe any one of them doesn’t work on most people but most people are susceptible to someone, and if you just take the convex hull of human persuasion ability you get something superhuman that can do the “normal talking but magically I agree” thing most of the time.
Like, given the existence of very unusually effective human persuaders, I just don’t see how someone could reasonably believe that superpersuasion-by-talking isn’t possible, unless they’re doing the motte-and-bailey that Zvi calls out by thinking that it has to mean ‘being able to persuade literally anyone of literally anything, no exceptions’ (I have seen exactly this a few times).
(Scott Alexander had a good post making this point, with Hitler(?), Joseph Smith, and Muhammad as examples, that I can’t find.)
Except there has been no progress towards this kind of “superpersuasion” since about GPT-4o despite impressive progress in benchmarks otherwise. If anything, there has been a regress because many humans on the Internet became more attentive to the signs of AI text (and thus more inclined to ignore unsolicited AI attempts to persuade).
The reason is quite obvious to me: there’s no scalable way to measure how persuasive was a certain LLM response, and thus it’s impossible to hill-climb this skill with post-training (and it doesn’t come for free with pre-training either). Note that social media reach and similar metrics don’t substitute for that
The taboo on ‘intelligence’ is not that people are just outraged by any attempts to talk about it; rather, there’s a general assumption that someone stating that one group of people is more intelligent than the other also believes that the former group is more deserving of moral consideration and political representation.
If I wanted to talk about differences in intelligence between races, I’d qualify it with the statement that all people deserve equal moral consideration and (upon reaching adulthood) representation. I also don’t want to talk about it, because I think all the evidence on it sucks and the people who talk about it rarely consider environmental contamination or socio-cultural factors, but I wouldn’t be offended if you asked me in that manner.
If someone disagrees with or omits that statement, then I’m going to assume they’ve been using motivated reasoning to affirm their previous biases, and they should be prepared to defend their sources beforehand. I am one of the people who make that assumption.
except that we have actual evidence that this is the case …
When Large Language Models are More PersuasiveThan Incentivized Humans, and Why, 2025
https://arxiv.org/abs/2505.09662
LLMs are literally more persuasive than people paid to argue the point. that’s a superhuman ability
Doesn’t say how the human persuaders were selected, and if they weren’t top experts it doesn’t demonstrate superhuman ability.
models also out persuade national championships debaters and professional canvassers, although the models regress when their ‘quantity of facts presented’ is limited to the human norms.
It’s covered in the first dispatch here.
https://aistop.watch/p/shifting-perspectives
Current LLMs can not learn online, and I do not expect them to gain this capabilities any time soon. There are several technical and social reasons why I expect this. The current approach to AI is not really compatible with it, and it would probably be prohibitively computational expensive. AI researchers seem uninterested in studying the extant examples of learning (in animal brains). And finally, it scares the model makers by threatening their control over the model weights.
Sensory-motor control is another area that the current paradigm seems to be wandering off track.
ASI might be on the horizon, but they are not going to have the same mental faculties as humans.
Mammalian/human memory systems feel seamless to us, but neurological/psychological study of them shows they’re actually quite a complicated cludge of working memory, short-term, medium-term, and long-term episodic memory, procedural memory, traumatic memory, and so forth: there are probably something of the order of half-a-dozen-to-a-dozen subsystems working together in human memory/learning.
For LLMs, the capabilities people have been working hard on this for the last half-decade or so, and so far as widely-used memory/learning mechanisms we have: initial model training, retrieval-augmented generation and memory summarization, and in-context learning plus context compaction. That’s not nothing, and each of them is improving, but the combination still has a distinct weakness in the sort of on-them-job learning that a lot of human workers do, often even during a specific extended project.
At this rate of advance, it seems unlikely to me that we’re going to devise a complicated architecture as capable as the human one in the next couple of years. So this makes me a little septical about some of the “full AGI in the next 2–3 years” claims: on those timescales I’m expecting “sort of AGI but with online learning less good than humans”.
I suggest a self-optimizer will use similar solutions if it helps further optimization, otherwise not.
Why do you call the “AI will be able to do more things than it can currently do” AGI? It’s certainly consistent to have an AI that can do a lot of things, but not everything a human can do.
I also think that ASI is different—I expect a literal country of sped-up geniuses to be able to do everything normal humans can do but better, but if we live in a world where magic prevents them from building a software ASI (e.g. physics is different enough that computers strong enough to host ASI are impossible), I don’t expect them to be literally able to take over the world, especially if there is a way for normies to read their thoughts not-to-inaccurately and give them orders that they mostly obey.
On the other hand, you could have an ASI that can take over the world if you let it do anything physical.
There is no AI better than gpt-4o / the model that summarizes Google search results [which was clearly false the day gpt-4o was released, since gpt-4 existed].
No AI significantly better than Mythos.
AI will get significantly better than Mythos, but still not obsolete humanity or be able to run the world by itself, even unopposed.
AI will plateau at a “country-of-genius” level, that can obsolete any mere mortal, but can’t quite take over the world if you use tricks to let it fight with itself.
ASI that can take over the world easily.