Long-time lurker (c. 2013), recent poster. Cunningham’s law is my friend.
For my own reference: (1) “benchmarks” very broadly construed (2) token consumptions & costs (3) satcat mass flow notes
Mo Putera
Karma: 3,590
Jacob Tsimerman, who’ll likely win the Fields Medal this year, thinks that AI “is boosting his productivity by a factor of two,” mostly by speeding up the boring parts.
Q: How does AI change math, the process or the feel of it?
A: Mostly it speeds up the boring parts. There’s the act of doing math, the professional endeavor, and the act of doing math as a fun endeavor. And I think AI helps the professional endeavor, and it gets you to the fun quicker.
… one of the primary ways that I use AI is to ask sort of dumb questions, or basic questions in fields in which I am not an expert. For example, I do a lot of research on Hodge theory. In a sense, I am an expert in Hodge theory because I’ve spent years thinking about it now. But there are still so many basics that I have to look up every single time, because I forget how the technical details work. In this way, I spend a lot of time during my research asking dumb questions, getting oriented.
Another example: I know how algebra works decently well. I’m used to working with rings. I’m used to working with schemes. I have some intuition there. Say my research needs me to work, as it did in the past, with -$p$adic rigid varieties or some analogous category, with continuous functions and their spectra, or whatever it is. Math has a lot of things that are kind of similar. And then I have intuition about how I’d hope different things might work in similar ways. Before Google, I’d have to go find experts, and they’d have to make time for me — I’d ask them my questions, and then go back and forth with them. With Google this was streamlined; I could look up books, search through them, get the basic theorems, try to put things together, learn the subject.
And now with LLMs, I just ask: “Hey, there’s this theorem in algebra. Does it basically work the same way in this other setting?”
Or: “Hey, if I have a group, and it’s this size, and it has such and such a property, does it also have this other property?”
You can plug that sort of thing into ChatGPT or Claude and it’s reasonably likely to give you something useful. And it can tell me the answer, and it can explain the answer. It can be wrong sometimes. There is a skill, a skill tree, in using it and figuring out when it’s bullshitting and when it’s wrong. But it’s immensely useful.
Q: With dumb questions as the starting point, where do you go next?
A: To break down the process, I would say there are a few different aspects of my math research workflow where AI comes in.
First, there’s the finding your bearing stage, getting oriented, where you’re figuring out what’s going on in your problem, or in your theory, whatever you’re grappling with. One part of that is determining what’s hard, what’s easy, what’s known, what’s not known. So, I’ll literally prompt the AI with: “Hey, I want to solve this kind of question. Give me an overview of what’s known and what’s hard.” And it will do that really well, consistently well.
And then I ask it more targeted questions, such as: “I was thinking of these special cases, or these kinds of analogs, which of these are known, and give me references.”
So that already is a huge time saver, partly because I can do it at scale. It’s much faster than asking a person, an expert; that would take, an email, waiting for a reply, or a meeting. Now I can try it a few different times immediately with the AI.
Second, there’s searching out a strategy, trying various types of arguments to see if they’re even in the realm of making sense. I can spell out my technique and ask if that’s been done before. I can ask, “What are the types of techniques people use?”
A third way it comes in handy is in looking up relevant material, and getting references, providing links — it’s gotten much better at providing links, but sometimes it will link to a website that doesn’t work anymore. I’ll prompt it with: “I want a result like this — is anything like this known? Please point me to references.”
And then fourth, there’s what most people call doing research. The fun parts of it, the real parts. You’ve spent months getting ready, you’re uploaded, you know what’s going on. There’s nothing left to look up. Now you have to think, and you have to come up with the right math. This is where flashes of brilliance happen, or just regular math work, whatever you want to call it. But all the fun parts of math are done here, where you’re just engaged with the problem. When I say my productivity doubles, it’s with everything leading up to this fun step.
Also:
Q: Do you think there will still be a human role in terms of ideas and creativity?
A: I think there will be a point where AI will be strictly better than humans at all aspects of math: learning, proving, coming up with the problems, aesthetics. It will just be better at everything. And this will come pretty soon,
Q: How soon is pretty soon?
A: Five years? I think in two years it might already be better than us at proving stuff. We’ll be able to say to it: “Here is a statement, go prove it.” But I have wide bars of uncertainty around this stuff. It’s hard to predict the future.
Curious what you found out?
I’m assuming your project is a continuation of this?
LW regularly surfaces years-old posts in my “For You” feed, most of which I haven’t read, with a surprisingly high hit rate of “I’d like to check this out”, e.g. ten seconds of scrolling down shows John Wentworth’s 7 year old The Missing Math of Map-Making, a few more seconds of scrolling shows DirectedEvolution’s 5 year old Build Your Number Sense, etc. Kudos to the LW team and I wish other forums did this.
Philip Kerger, who teaches industrial engineering and operations research (IEOR) at UC Berkeley, just used GPT 5.6 Sol Pro in chat(!) to one-shot prove a problem in convex optimisation that had stumped him after sporadically working on it for a year or so, and that other researchers had tried and failed at (“only last year at a the ICCOPT conference I heard someone say “we have no idea” how to solve this”). Here’s the chat producing the initial proof after 148 minutes, follow-up chat refining it after 230 minutes, GitHub with Lean code and more, etc.
Kerger’s problem description from reddit
The problem concerns deterministic zeroth-order convex optimization: Let B_d be the Euclidean unit ball in ℝᵈ, and consider all convex, 1-Lipschitz functions f: B_d → ℝ. An algorithm may query any point x ∈ B_d, and receives only the exact real number f(x), no other information (but the algorithm “knows” that f is convex and Lipschitz). The algorithm is otherwise completely unrestricted, and can use unlimited computation and memory. These function-value-only problems arise naturally when an objective is evaluated through a physical experiment or simulator. One can imagine choosing d engineering parameters and observing only the cost returned by the simulation. If evaluations are expensive (think of measuring a physical system), the natural question is how many are fundamentally required. This is formalized as oracle complexity. Specifically, this is the oracle complexity of convex optimization under an exact function value oracle.
Let Q(d, ε) denote the worst-case number of queries required to find an ε-optimal point of f. An algorithm due to Protasov from 1996 shows that order d² function evaluations are sufficient, which gives Q(d, ε) = O(d²), an upper bound on the complexity. Lower bounds were practically nonexistent for this setting, and the strongest previously applicable bound was only Ω(d), inherited from the stronger first-order oracle model (where the algorithm receives both function values and gradients). That means we didn’t know for certain whether gradients actually help in optimization, since the function-value only and first-order oracle models have had this same lower bound, and so there was a linear gap in d in the complexity of this fairly fundamental convex optimization setting since 1996. So, can you find an algoritm that is better than Prosatov’s, and only needs d evaluations? Or can you show that no such algorithm can exist, and we can sleep well at night knowing that Protasov’s algorithm using d² evaluations is best possible? What 5.6 Sol proved is the latter.
What caught my eye was Kerger’s 10-page(!) effortprompt, available in Appendix A.1 of his paper, inspired by but significantly longer than OpenAI’s own effortprompt behind their proof of the cycle double cover conjecture:
Snippets of his effortprompt
Page 4 -- vibes-wise this reminded me of Eliezer’s old post referencing The Open-Source Wish Project, Wish For Immortality 1.1
Final page:
How did he come up with such a prompt? With Sol’s help of course
Kerger (paragraphs mine):
I basically had 5.6 Sol synthesize existing closely related work and their approaches, the past ideas I had, with OpenAI’s prompt that had a lot of the presumably important mechanisms for how exactly the agent should act. Especially from the “results that do not count” section onwards is a lot of input from Sol.
A major reason for the added length though is also the nature of the problem; Just the description of the problem and pointing to a couple existing results already put me at a couple pages. With this kind of complexity result what exactly fulfills the specifications of what is needed can be nuanced (so later there is also a lot more of “what doesn’t count”, and there are two major components to any proof namely the function class to use and the adversarial oracle strategy, about which you then need to prove things with convex geometry machinery, so that all adds to the length).
Also, this gap could have in principle been closed from either direction, which also adds a bit (though you can probably read between the lines that I was trying to push much more for a lower-bound, since an algorithm matching order d complexity would have been extremely surprising, since that would mean you can optimize without gradients just as fast as you optimize with them, which would be shocking given that so much practical optimization happens with gradients)
5.6 Sol was a lot better than 5.4 and 5.5 at this, although the effortprompt probably helped a lot
when I was working with 5.4 and 5.5., it kept returning to me with “Here’s great progress! The only step left is to prove this one lemma here, and here’s why proving that lemma would get the result”. Then I’d dive into trying to prove that lemma together with 5.5 and it would go nowhere. I’ll share here a chat with you that I had with 5.5 Pro Extended, where I actually told it to use max of affine functions as the hard function class (which is what ended up working and is in the preprint): https://chatgpt.com/share/6a592503-2d60-83ea-8f00-9ed96b331b16
it thought for a whole 4 minutes in its initial response. But, my prompting was also missing a lot of the what counts as a solution, only return when you have found definiteve answers, explore lemmas like this, and so on. Disclaimer, I was a bit frustrated at this point in time and was just trying to get the model to dive deeper on a couple of different approaches in sessions I was running in parallel, so most of my replies there are basically just “keep going”, especially since I had already told it what I want it to do in the initial prompt.
It cost him 10% of his $200 Pro plan (with the initial 5h/day cap)
Kerger (paragraphs mine):
Between $20 to $200 depending for me depending on how you want to look at it. I got the $200/month subscription specifically to try tackling some problems together with AI.
This specific project used under 10% of my weekly limit on there (though OpenAI messed with limits, increasing them, during this project, so I’m not sure if that’s 100% accurate). Total time was about 2.5h 5.6 Sol Pro use on the initial proof outline, 4h 5.6 Sol Pro on an improvement on the accuracy requirement, ~2h use of verifications and proof audits, ~4h use assisting for the Lean development (these later ones are estimated, and only the time of the actual active tool use, whereas the total time including my own work was of course much more). So something on the order of no more than 15h of actual time.
Even with the initial 5h cap per day they had (and then lifted), that uses 3 days = 10% of the monthly pro plan, so from the user side I can in some way say about $20 (though probably OpenAI’s compute cost much more than that!), or the $200 since I did get the pro plan for this.
As a point of reference, it cost Google DeepMind’s Gemini 3.1 Pro-based agent a few hundred dollars per proof of the 9 open Erdős problems it autonomously solved out of the full set of 353 in the open-source Formal Conjectures repo, with predictably very large variance:
As before, Sol’s proof didn’t really use nor create new techniques:
Lastly, some important comments about the work relating to AI capabilities: In a lot of cases, proving lower bounds like this result relies on finding that right construction that works (in this case, family of difficult functions and a strategy for how an “adversarial” oracle should answer queries from an algorithm to reveal minimal information) and then proving things about it. There are only so many function classes which would be reasonable to look at (here, quadratics for example would have also been reasonable with order d² degrees of freedom, or any variation of maxes of some simpler families of convex functions as well), but the actual proof mechanics once the “correct” function class and correct strategy for adversarial oracle answers is found are often not so complicated, and often employ existing results from convex geometry or similar (this is also the structure of two previous but much more niche, less important results of mine). So I wouldn’t really say that this result is using or creating some fundamentally new techniques in convex geometry or optimization theory.
What this means from my perspective is that if a result is attainable with existing techniques, modern AI methods will be able to solve those problems. I don’t think researchers in math/TCS will be made obsolete, but I think it will instead no longer make sense to work on any low-hanging, or even medium-hanging (you know what I mean) fruit. We’ll be needed for problems where actual novel approaches are needed.
But frankly I don’t think this is needed for even Fields-level contributions. To quote my own quick take:
Think about what a (static!) automated Jean Bourgain-toolkit interpolator could do
What’s the most impressive research-y feat interpolating AIs can theoretically do, fixing their training data to (say) today?
I don’t have a good sense of this in general, but in pure math it’s probably at least Fields medal-tier, if laudatios like that of Akshay Venkatesh are anything to go by:
Akshay Venkatesh stands out for the startlingly original way he has connected number theory problems to deep results in other areas. Far from using them as “black boxes” to crank out solutions, Venkatesh brings fresh insights to the results and highlights their unexpected connections to number theory. In this way he has made striking advances in number theory while also greatly enriching other branches of mathematics.
This, and the rest of the laudatio, reads like a very souped-up version of what OpenAI’s recent internal model did with Erdos problem #90 (itself a souped-up version of what GPT-5.4 Pro did with problem #1196), give or take a few Gowers-hints. One of Venkatesh’s advantages over other top-tier mathematicians is his sheer range, the thing frontier models do vastly better than humans at.
And I can imagine, for instance, 2 years of advancements enabling a frontier model to skillfully deploy the late legendary Jean Bourgain’s toolkit. Bourgain was regularly spoken of by other world-leading mathematicians as “effectively a god”. Terry Tao:
When I was a graduate student in Princeton, Tom Wolff came and gave a course on recent progress on the restriction and Kakeya conjectures, starting from the breakthrough work of Jean Bourgain in a now famous 1991 paper in Geom. Func. Anal.. I struggled with that paper for many months; it was by far the most difficult paper I had to read as a graduate student, as Jean would focus on the most essential components of an argument, treating more secondary details (such as rigorously formalising the uncertainty principle) in very brief sentences.
Tao goes on to describe Bourgain’s style and toolkit:
I began to realise that Jean had a certain collection of tools, heuristics, and principles that he regarded as “basic”, such as dyadic decomposition and the uncertainty principle, and by working “modulo” these tools (that is, by regarding any step consisting solely of application of these tools as trivial), one could proceed much more rapidly and efficiently. By reading through Jean’s papers, I was able to add these tools to my own “basic” toolkit, which then became a fundamental starting point for much of my own research. Indeed, a large fraction of my early work could be summarised as “take one of Jean’s papers, understand the techniques used there, and try to improve upon the final results a bit”.
In time, I started looking forward to reading the latest paper of Jean. I remember being particularly impressed by his 1999 JAMS paper on global solutions of the energy-critical nonlinear Schrodinger equation for spherically symmetric data. It’s hard to describe (especially in lay terms) the experience of reading through (and finally absorbing) the sections of this paper one by one; the best analogy I can come up with would be watching an expert video game player nimbly navigate his or her way through increasingly difficult levels of some video game, with the end of each level (or section) culminating in a fight with a huge “boss” that was eventually dispatched using an array of special weapons that the player happened to have at hand.
Imagine unleashing thousands of Bourgain-toolkit interpolators math-wide in 2028. I think I’m being conservative here, not assuming continual learning or whatever, not even assuming anyone else’s toolkit, and yet I still find it hard to imagine how transformative this would be. And this is just for pure math.
Look. I’m the last person who’s going to deny that the road we’re on is littered with the skulls of the people who tried to do this before us. But we’ve noticed the skulls. We’ve looked at the creepy skull pyramids and thought “huh, better try to do the opposite of what those guys did”. Just as the best doctors are humbled by the history of murderous blood-letting, the best leftists are humbled by the history of Soviet authoritarianism, and the best generals are humbled by the history of Vietnam and Iraq and Libya and all the others – in exactly this way, the rationalist movement hasn’t missed the concerns that everybody who thinks of the idea of a “rationalist movement” for five seconds has come up with. If you have this sort of concern, and you want to accuse us of it, please do a quick Google search to make sure that everybody hasn’t been condemning it and promising not to do it since the beginning.
We’re almost certainly still making horrendous mistakes that people thirty years from now will rightly criticize us for. But they’re new mistakes. They’re original and exciting mistakes which are not the same mistakes everybody who hears the word “rational” immediately knows to check for and try to avoid. Or at worst, they’re the sort of Hofstadter’s Law-esque mistakes that are impossible to avoid by knowing about and compensating for them.
And I hope that maybe having a community dedicated to carefully checking its own thought processes and trying to minimize error in every way possible will make us have slightly fewer horrendous mistakes than people who don’t do that. I hope that constant vigilance has given us at least a tiny bit of a leg up, in the determining-what-is-true field, compared to people who think this is unnecessary and truth-seeking is a waste of time.
David Reinstein’s tool might be of interest to readers https://uj-ai-wealth-philanthropy-steelman.netlify.app/
This is getting very tangential, but the moment I saw “Xiaoguang Li” I thought of the thousand year old vampire quote:
I once lent Xiaoguang “Mike” Li my copy of “Probability Theory: The Logic of Science”. Mike Li read some of it, and then came back and said:
“Wow… it’s like Jaynes is a thousand-year-old vampire.”
Then Mike said, “No, wait, let me explain that—” and I said, “No, I know exactly what you mean.” It’s a convention in fantasy literature that the older a vampire gets, the more powerful they become.
Yeah, to corroborate your 1st bullet point, I came from the private sector where I spent over a half decade as a data analyst worrying about and ensuring trustworthiness of numbers reported to executive teams making lots of tight-feedback loop high-stakes decisions, and when I pivoted to my current academia-adjacent career path I was shocked to see how much worse data trustworthiness was in comparison even for supposedly high-quality journal articles when I started digging into things.
They were:
Some proof cost data from ProofBench, h/t Logan Riggs. Fable is faster and cheaper than the others.
I cannot confidently say that this was faster than doing research the old-fashioned way. Sitting down with AI puts my attention in very different places, more on intuitions and less on the math; hard to say how quickly I could have arrived at similar results myself. What I can say is that Claude 4.8 is over some sort of tipping point for me, where I feel like I can “just keep going and keep making progress” in some new sense. I’m sure Fable would have done a better job, and the quality of vibe research will keep getting better from here
Reminds me of what Terry Tao said, emphasis mine
Dwarkesh Patel
So let’s see if you can continue this streak. You personally are 2x more productive as a result of AI. What year would you say that?
Terence Tao
Productivity, I think, is not quite a one-dimensional quantity. I’m definitely noticing that the style in which I do mathematics is changing quite a bit, and the type of things I do. For example, my papers now have a lot more code, a lot more pictures, because it’s so easy to generate these things now. Some plot which would have taken me hours to do, now I can do in minutes. But in the past, I just wouldn’t have put the plot in my paper in the first place. I would just talk about it in words. So it’s hard to measure what 2x means.
On the one hand, I think the type of papers that I would write today, if I had to do them without AI assistance, would definitely take five times longer. But I would not write my papers that way.
Dwarkesh Patel
5x?
Terence Tao
Yeah, but these are auxiliary tasks. Things like doing a much deeper literature search or supplying a lot more numerics. They enrich the paper. The core of what I do, actually solving the most difficult part of a math problem, hasn’t changed too much. I still use pen and paper for that.
But there’s lots of silly things. I use an AI agent now to reformat. Sometimes if all my parentheses are not quite the right size, I used to manually change them by hand, and now I can get an AI agent to do all that quite nicely in the background.
They’ve really sped up lots of secondary tasks. They haven’t yet sped up the core thing that I do, but it’s allowed me to add more things to my papers. By the same token, if I were to write a paper I wrote in 2020 again—and not add all these extra features, but just have something of the same level of functionality—it actually hasn’t saved that much time, to be honest. It’s made the papers richer and broader, but not necessarily deeper.
The anecdotes from this thread on r/careeradvice seem like gradual (self-)disempowerment:
OP:
I’m a senior executive with almost 2 decades of experience both at startups and large tech companies. Most recently I joined a VC backed startup right before the AI boom started to drive GTM and sales. It is a small company and I report to the CEO. …
My biggest issue is that I am getting increasingly frustrated with the CEO’s constant reliance on AI for everything. It is honestly like working with “Claude as the CEO”. There are literally copy-paste passages as “strategy” that Claude spits out on slack as guidelines. There is not even an effort to hide or think through it anymore. I use it for work too but I see it as a tool to help refine my thoughts—not tell me how to run a business.
To someone’s suggestion that they share their concerns with a board member they trust, they replied:
I’ve worked for startups before. I did my due diligence before signing up for this one. The CEO is actually board appointed and not the founder. He is quite experienced. We worked well together at the beginning but over the past 6-8 months it has progressively gotten worse.
I don’t see going to board as being the right choice. Specially given the slow growth of the business.
Another commenter seeing their CEO become a Claude messenger:
I also report to the CEO and there were a couple times I asked her a question for input and got copy pasta from AI. The second or third time I was like, “Look, I can ask AI myself, but I asked you.” It prompted a further discussion about how I’m asking her to make sure we’re covering both the expectations of her and the client and give her an opportunity to tell my any information she has in her head that she hasn’t shared yet, since I wasn’t a party to a lot of the conversations, and I can pair those with AI suggestions myself. She stopped.
Another commenter talking about their supervisor’s behavior being in keeping with other “millennials and boomers”:
Worked in the strategic finance team at one of the largest software companies in world. My team mostly filled with millennials and boomers (50s+).
I am really pissed on how most of them use AI tools, my supervisor used claude like religion, where she basically generated everything out of it, from email replies, reports, and even strategic presentations. And she openly admitted that all of it were AI generated in the meeting room and nobody really cares to check it, they just agreed to all of it quickly.
And I really hate it when they thought that AI results are words from Almighty and always 100% correct. These people should learn more on how to use AI.
I am calling these people’s as “servants of AI.”
Another commenter talking about senior folks:
Seeing this in other roles where a senior or lead is essentionally doing zero review, fact checking, scrutininy, quality assurance, critical thinking, analysis etc. of AI output.
Then, calling it out is seen as not adopting AI or you get tagged in to fix everything. You also get the pleasure of being seen as antagonistic for barely hanging on to your sanity while most people around you do not even try to make it look they they are doing their fucking jobs.
You are also elitist or racist for correcting grammar on public facing work content, spelling mistakes or suggesting we don’t need to use the same recycled jargon.
You should generally downgrade reddit anecdotal evidence, but I’m seeing increasingly more examples of this myself unfortunately so they all rang true/plausible.
Snippets from the Anthropic Economic Index June 2026 report, emphasis mine:
Our survey allowed us, for the first time, to ask people directly about how they use AI and what they feel about it. We found that our survey respondents use AI for more than we give it credit for—they report AI can do a higher share of their work than the observed exposure measure for their occupation would suggest. Asked to forecast next year’s capabilities, over 35% predicted that AI would be able to do most of their work.
… the best-fit lines for reported and anticipated exposure 12 months from now (orange dots) are essentially parallel, meaning that people in roles with high observed or theoretical exposure expect roughly the same increase in the share of their work tasks AI can do over the next year as those in roles with less observed and theoretical exposure.17 In other words, a software engineer and a construction manager anticipate roughly the same increment of progress within their profession.
It is also worth noting that reported exposure systematically exceeds observed exposure. One explanation for this is that not everybody does every task in an occupation, and our survey disproportionately reaches those who use AI more.18 Analogously, since theoretical exposure is an upper bound on what is possible instead of a measure of current use, theoretical exposure systematically overstates reported exposure.
… perceptions of AI’s capabilities are negatively correlated with country GDP:19 The average share of tasks people report AI can do for them now is about 10 percentage points lower among high-income countries. This pattern is consistent with the possibility that AI substitutes for a larger share of the tasks that workers in lower-income countries do day-to-day, even if occupation-level exposure metrics—which tend to be higher in advanced economies—suggest otherwise.
… we study who uses Claude in various ways. The most striking differences are by gender. Women, who make up only 12% of our linked respondent sample, use Claude differently from men. Even after accounting for occupational differences, they are marginally less likely to use Claude for work, their share of sessions in Claude Code is 0.24 standard deviations lower (6.3 percentage points), and their automation share is 0.33 standard deviations lower (7.3 percentage points). Instead, women tend to use Claude more iteratively, and they log more active time on chat than men, a signal of more collaborative engagement.27
The survey respondents:
In 2022 Metaculus ran Forecasting Our World in Data: The Next 100 Years, still the most methodologically thorough metrics-grounded whole-of-civilisation forecast of its kind I know of:
Metaculus has forecasted the trajectory of 30 Our World in Data metrics on its platform using both a public tournament and a group of Pro Forecasters. Both accuracy and transparency in reasoning were considered essential to the project. The public tournament was available to the platform’s full community of over 2,000 forecasters, while the private forecasting space was intended for a small group of the top 2% of Metaculus’s most accurate forecasters. Combining both approaches allows for a variety of perspectives and a broad discussion, as well as ensuring that predictions are of the highest quality. With substantive discussions surrounding this tournament’s 30 forecasting questions, Metaculus has been able to better understand: (1) approaches and methods adopted by elite forecasters, (2) rationales driving their prediction for each time period, and (3) a composite perspective into our world 100 years from now. …
Based on the timeframes that were of greatest interest to our stakeholders, Metaculus elicited forecasts for a variety of intervals — 1, 3, 10, 30, and 100 years out from the year 2022. For each OWID metric that Metaculus and the initiative’s funders selected, this tournament created grouped questions that asked forecasters to give their predicted value of that metric in the years 2023, 2025, 2032, 2052, and 2122. With such an array of time horizons, the tournament takes a less common forecasting approach that allows for direct comparison between short-term and long-term predictions.
The composite perspective into the world of 2122 seems distinctly pedestrian to me
Our World in 2122
In the year 2122, the global population is expected to be around 8 or 9 billion, with a chance that number reaches as high as 20 billion due to technological and economic advancements. Lifespans are expected to be significantly longer, especially in the G7 countries, thanks to the achievement of longevity escape velocity. Reproduction is anticipated by some to primarily utilize ectogenesis technology, though some may choose to avoid it for personal or ethical reasons. The decline in population is expected to slow, but there remains uncertainty as to whether it will level off or decrease further.
On the economic front, GDP per capita is believed to be quite high and will likely continue increasing. In fact, most economies are expected to be as advanced as present-day Scandinavian countries, with energy being inexpensive and the price of most goods and services being close to zero. Productivity is forecasted to significantly increase, with advanced artificial intelligence playing a major role in GDP growth. The United States economy is anticipated to continue growing, potentially transitioning to a more service-based industry and experiencing changes in government structure. However, the concept of the United States as we know it may be fundamentally different by this time. Instead, it is possible that global governance will be implemented and the number of sovereign states will drastically decrease. Wealth distribution remains a key uncertainty — while it’s possible that a small group could control a significant portion of resources, a shift toward more equitable distribution may win out.
Even with global governance, many pressing challenges will remain. Despite drastic cuts in carbon dioxide emissions, climate change is expected to be a major concern — sea levels will rise and extreme weather events will become more frequent. Nevertheless, renewable energy is also expected to increase in parallel, with the potential for advanced technologies, like nuclear fusion, to power a significant portion of the global economy. Even though the precise global energy mix remains uncertain, transportation will undergo major electrification, with an anticipated shift towards both zero emissions and autonomous vehicles.
Technology will continue expanding at a rapid pace and play an even greater role in daily life. The use of robots is predicted to increase and revolutionary innovations like brain-computer interfaces could fundamentally change daily life as we know it. Education is therefore expected to become more personalized and virtual, as will the healthcare system. The latter will undergo many transformations as artificial intelligence opens the door for hyper-personalized treatments and precision medicine. The concept of work will change too, becoming far more flexible and remote. Leisure time is expected to increase, with virtual reality and space tourism becoming popular for many.
Despite great uncertainty and persistent challenges, 2122 is expected to be a time of great advancement and possibility.
I wondered how the first two timeframes (2023 and 2025) had done so far, so I asked the models:
GPT-5.5 (high): “pretty good on slow-moving macro and demographic variables, good on space objects and CO₂, and bad on a few discontinuous or definition-sensitive variables: terrorism, fastest-supercomputer FLOPS, people living in liberal democracies, and AI training compute”
I used the Metaculus report tables for the forecast medians, with “Pro” meaning Pro Forecaster median and “Public” meaning the Metaculus Prediction. The report says the tournament forecasted 2023, 2025, 2032, 2052, and 2122 values, and that the summary tables cover the 20 questions forecasted by both Pros and the public.
Bottom line
So far, the forecasts look pretty good on slow-moving macro and demographic variables, good on space objects and CO₂, and bad on a few discontinuous or definition-sensitive variables: terrorism, fastest-supercomputer FLOPS, people living in liberal democracies, and AI training compute. AI investment is messy: they got the direction right, but whether the forecast was high or low depends heavily on whether you use narrow private funding, broad corporate AI investment, or capex-like spending.
A rough scorecard: among the items I could actually check with reasonably current data, I’d call about 7 good or very good, 4 clear misses, and the rest not yet properly scoreable because the official annual data lag into 2026 or 2027.
Scorecard for the clearest cases
Metric
Forecast for 2023 and 2025
What happened
Verdict
Annual objects launched into space
2023: Pro 3,000, Public 2,700. 2025: Pro 4,300, Public 4,000.
OWID says 2,903 objects were launched in 2023 and 4,510 in 2025.
Very good. Pros were especially close. Public undercalled 2025 by about 11%.
Global CO₂ emissions
2023: Pro 37.7 Gt, Public 36.4 Gt. 2025: Pro 38.1 Gt, Public 36.2 Gt.
IEA put 2023 energy-related CO₂ at 37.4 Gt; Global Carbon Project projected 2025 fossil CO₂ at 38.1 Gt.
Pros basically nailed it. Public forecasts were too optimistic about emissions falling.
Fastest supercomputer FLOPS
2023: Pro 1.5e18, Public 1.3e18. 2025: Pro 4.1e18, Public 4.5e18.
Frontier was still around 1.1 exaFLOP in late 2023; El Capitan led the Nov. 2025 TOP500 at 1.809 exaFLOPS Rmax.
Clear miss high. They expected the exascale transition to accelerate faster than the TOP500 benchmark actually did.
Global nuclear warhead stockpiles
2023: Pro 9.46k, Public 9.40k. 2025: Pro 9.52k, Public 9.38k.
SIPRI estimated about 9,614 warheads in military stockpiles in Jan. 2025.
Very good. Slightly low, but right order and right direction.
Total terrorism fatalities
2023: Pro 19k, Public 20k. 2025: Pro 19k, Public 20k.
Global Terrorism Index reported 8,352 deaths in 2023, and its 2026 report gives 5,582 deaths in 2025.
Bad miss high. They overestimated by roughly 2.3x in 2023 and 3.4x in 2025.
People living in liberal democracies
2023: Pro 1.02B, Public 1.05B. 2025: Pro 1.00B, Public 1.10B.
V-Dem’s 2026 Democracy Report says only 7% of the world population, about 0.6B people, lived in liberal democracies at end-2025.
Clear miss high. This is probably the biggest substantive miss, driven by democratic backsliding and classification changes, especially the U.S. losing liberal-democracy status in V-Dem.
Greatest AI training compute
2023: Pro 6.5e24 FLOP, Public 1.3e25. 2025: Pro 2.9e25, Public 6.6e25.
Epoch’s 2026 trends page says the largest known training run was Grok 4 at around 5e26 FLOP.
Miss low. They correctly expected rapid growth, but still undercalled the frontier compute boom by nearly an order of magnitude versus the public 2025 median.
AI company investment
2023: Pro $195B, Public $282B. 2025: Pro $340B, Public $489B.
OWID’s AI investment series is narrow external private-market funding and explicitly excludes Big Tech internal R&D, capex, public-sector funding, and publicly traded firms; Stanford’s 2026 AI Index says private investment grew 127.5% in 2025 and was 60% of total corporate AI investment.
Directionally right, hard to score. On broad corporate investment, the public forecast may be close. On narrow private-company external funding, the forecasts look high.
Global fertility rate
2023: Pro 2.29, Public 2.23. 2025: Pro 2.24, Public 2.20.
OWID says global total fertility was 2.3 children per woman in 2023; UN’s 2024 fertility report gives a 2024 estimate of 2.2.
Good. The public 2023 forecast was a little low; the 2025 forecast is probably close, though final 2025 data are not fully settled.
World population
2023: Pro 8.04B, Public 8.06B. 2025: Pro 8.18B, Public 8.21B.
UN WPP 2024 gives estimates and projections through the present; World Bank summarized 2025 as about 8.2B people.
Good. Slightly low, but well within the range you would expect for this kind of forecast.
Same-sex marriage legal share
2023: Pro 15.5%, Public 15.3%. 2025: Pro 18.0%, Public 16.0%.
OWID says 1.5B people, about one in five globally, lived where same-sex marriage was legal in 2025; Thailand and Liechtenstein were added in 2025.
Good, especially Pros. Their expectation of modest continued expansion was right.
OECD trust in government
2023: Pro 42%, Public 43%. 2025: Pro 42%, Public 41%.
OECD’s 2023 survey found 39% with high or moderately high trust in national government across surveyed OECD countries. OWID’s Gallup-based OECD series is updated only through 2024.
Slightly high, not disastrously. This is not yet a clean 2025 score.
Per-capita primary energy consumption
2023: Pro 21,200 kWh, Public 21,100. 2025: Pro 21,600, Public 21,400.
Energy Institute reported global primary energy consumption at 620 EJ in 2023; with world population around 8.09B, that implies about 21,300 kWh/person. OWID’s series is updated through 2024.
Very good for 2023. 2025 not cleanly scoreable yet.
Things that are still annoyingly unresolved
Several forecasts are “past” in calendar time but not actually resolvable in the official-data sense. This matters because OWID-style indicators often update with a one- to three-year lag, and some of the underlying data series have changed methodology.
Metric
Status
Chickens slaughtered for meat
The forecast was 75.9B/74.7B for 2023 and 79.6B/78.2B for 2025. But global animal-slaughter data through FAOSTAT/OWID lag badly; OWID’s recent public material still discusses 2022 levels around 83B total land animals, mostly chickens. I would not score 2025 yet.
Cost of sequencing a whole human genome
Forecasts were $357/$303 for 2023 and $244/$191 for 2025. NHGRI’s canonical cost series explains the methodology and downloadable table, but the page still points to “Sequencing Costs 2022,” so this is not cleanly resolvable from the original benchmark.
Global poverty below $10/day in 2011 international dollars
Forecasts were about 4.66B in 2023 and 4.61B in 2025. World Bank poverty lines and PPP vintages have since shifted, making this surprisingly nontrivial to score without reconstructing the exact old line and PPP basis.
G7 life expectancy
Forecasts were 81.4/81.2 in 2023 and 81.7/81.1 in 2025. OECD data show recovery from COVID mortality, with OECD average life expectancy at 81.1 in 2023 and provisional 2024 increases, but I did not find a clean population-weighted G7 actual for 2025.
Global homicide rate
Forecasts were 5.24/5.32 in 2023 and 5.14/5.26 in 2025. World Bank’s homicide metadata runs through 2023, and the 2023 value looks close to their forecasts, but 2025 is not available.
US GDP per hour worked
Forecasts were $77.60/$76.60 in 2023 and $79.60/$78.40 in 2025. OWID’s Penn World Table source is updated only through 2023, and the U.S. BLS productivity series is not the same dollar-level metric. Directionally, 2023-24 U.S. productivity growth was stronger than many expected.
GDP per-capita growth
Forecasts were 3.1%/2.8% in 2023 and 2.8%/2.6% in 2025. OWID/World Bank annual GDP-per-capita growth is updated through 2024, while the IMF projected 2025 global GDP growth at 3.2%. Since population growth was under 1%, 2025 real per-capita growth was probably lower than the Metaculus medians, but I would score this only after the final World Bank/OWID series updates.
My read
The forecasts were not embarrassing. The strongest thing about them is that the “boring” forecasts mostly worked. Population, fertility, nuclear stockpiles, CO₂, energy use, space objects, and same-sex marriage legality were all broadly on track. The forecasters were doing something reasonable: anchor to recent series, adjust for obvious shocks, and avoid inventing too much discontinuity for 2023 or 2025.
The misses are more interesting. They overexpected institutional continuity in democracy, overexpected official supercomputer benchmark growth, badly overexpected terrorism fatalities, and underexpected the scale of AI training runs. That combination is telling: Metaculus was good at smooth continuation and at already-visible deployment trends, but weaker when the outcome depended on classification thresholds, geopolitical/security-event volatility, or private frontier-lab spending.
The AI cases split in opposite directions. For training compute, the 2025 forecast was too low because the post-ChatGPT capex frenzy and frontier-lab race got hotter than the early-2023 forecasters priced in. For AI investment, the answer depends on the denominator: narrow external private funding makes some forecasts look high, while broad corporate AI investment makes them look much more reasonable. The report’s own AI-investment metric is therefore less clean than it first appears. (Our World in Data)
Overall, I’d give the 2023/2025 track record something like B or B-minus if judged across all 20 topics, and B-plus on slow-moving OWID-style metrics. The important caveat is that this is not a formal scoring exercise over full distributions. It is a median-vs-actual audit, and for about a third of the metrics the “actual 2025 value” is still not fully published or has become methodologically slippery.
Opus 4.8 (max): “excellent on everything that was structurally predictable and wrong in exactly the places where the institutional consensus they anchored on was also wrong — which means the near-term scorecard mostly grades the 2022–23 consensus, not the forecasters’ distinctive skill”
My plan: this is fundamentally a forecasting-calibration question, and per the analyse skill I should earn the escalation — spend search budget disproportionately on the metrics where a regime change could have broken the forecast (AI compute, AI investment, supercomputers, CO2, space launches, same-sex marriage), and merely spot-check the slow-moving demographic/physical series (population, fertility, nuclear, sequencing cost) that 1–3-year extrapolation almost always nails. The information is in the surprises, not the confirmations.
I have what I need across all 20 Pro-forecasted metrics. One honesty caveat before the verdict: the report gives me the forecasters’ described reasoning and some point numbers, but the precise Pro medians and interquartile ranges live in charts I can’t see. So for the clear cases (AI investment ~6×, terrorism +22% vs. a predicted decline, CO₂ records vs. “at peak”) the direction is unambiguous regardless of the exact median; for the “hit” calls, “landed in range” is my inference from the described forecast sitting close to the actual, not a certified percentile. I flag where that matters.
Scorecard — the 20 Pro-forecasted metrics (2023 & 2025)
Sorted most-informative first (misses, then measurement problems, then the quiet hits).
Metric
Near-term forecast (as described)
What actually happened
Verdict
AI training compute
Doubling ~10 mo (≈2.3×/yr)
GPT-4, first model >1e25 FLOP, March 2023; Gemini Ultra ~5e25 by Dec 2023; ~1e26 by late-2025, ~4.7×/yr
Miss (under) — real growth ~2× their rate
AI investment
“Divided,” modest growth, some expecting a slowdown
From $91.9B private in 2022 → $252.3B corporate in 2024 → $581.7B corporate / $344.7B private in 2025
Miss (under) — biggest miss in the set
Terrorism fatalities
Continue to decline in 2023
Rose 22% to 8,352 in 2023, highest since 2017 (Oct 7 + Sahel); declined again in 2024
Miss — black-swan (Oct 7); they’d flagged it as noisy
GDP growth per capita
Poor 2023, recession in many advanced economies
Soft landing; the feared recession didn’t arrive; world grew ~2.95% in 2023
Miss (too pessimistic) — anchored on IMF/WB gloom
CO₂ emissions
Number ~37–38 Gt; “at or near peak”
36.8 Gt (2023) → 37.4 Gt (2024), record, “no sign of peak”
Split — number nailed, peak narrative premature
Genome sequencing cost
~15%/yr decline; $100 genome ~2030
NHGRI/OWID series frozen at 2022; commercially $200 (Illumina, 2024) and $100 (Ultima, 2024)
Unresolvable — data pipeline died; tech beat the timeline
Objects launched to space
2025 range 3,900–9,200
2,903 in 2023; 4,510 in 2025
Hit — actual in lower part of range
Fastest supercomputer
Sub-2 EF in 2023; no 5–10× leap by 2025
Frontier ~1.2 EF (2023); El Capitan 1.809 EF (Nov 2025)
Hit
World population
~8.05B (2023), ~8.19B (2025), UN medium
~8.0B in 2023, ~8.2B 2025
Hit — essentially dead-on
Global fertility
Keep declining, slightly below UN
2.3 in 2023; 2.40 in 2025
Hit
Nuclear stockpiles
~recent level, slight change, China up
12,241 total (Jan 2025), down slightly; China 500→600
Hit
Same-sex marriage share
Modest rise; Thailand by ~2032
38 countries, 1.5B people, 20% by 2026; Thailand Jan 2025
Hit — Thailand arrived ~7 yrs early
US productivity
~2.5%/yr, above the ~1.1% pre-pandemic trend
+1.6% (2023), +2.3% (2024), ~2.1% (2025)
Hit — slightly high, acceleration call right
Per-capita energy
Slow increase
~77 GJ/person (2023); record totals, modest per-capita rise
Hit
Life expectancy (G7)
Slight rise; US continues to drag
82.0 avg (2024); US lowest at 78.9
Hit
OECD trust in govt
40–50%, slight dip
48% in 2023
Hit — mild pessimism lean
People in liberal democracies
Roughly flat / slight decline
V-Dem shows continued slow decline (~1.0B; classification contested)
Hit (directional) — metric is itself noisy
Chickens slaughtered
Continued growth
FAO data lags ~2 yrs; 2023 record, rising; 2025 N/A
Lagged — directionally on track
Global homicide rate
Continued slow decline
UNODC lags ~2 yrs; not yet resolvable
Lagged
Global poverty (<$10/day)
Continued decline
World Bank lags ~2–3 yrs; not yet resolvable
Lagged
Tally: roughly 12 clean hits, 1 split (CO₂), 4 misses, 1 measurement failure, 3 not-yet-gradable.
The pattern
Governing thought: at 1–3 years, these forecasts were excellent on everything that was structurally predictable and wrong in exactly the places where the institutional consensus they anchored on was also wrong — which means the near-term scorecard mostly grades the 2022–23 consensus, not the forecasters’ distinctive skill.
Three claims hold it up:
The misses are not random — they cluster into three buckets. Two of the four (AI compute, AI investment) are the same phenomenon: an exponential the entire analyst world under-priced in February 2023. One (GDP growth) and the pessimism lean visible in trust-in-government and the CO₂-”peak” call are the opposite error — over-weighting the salient 2022–23 gloom (the recession everyone forecast and that never came, the IEA’s perennial imminent-peak). One (terrorism) is a genuine black swan — Oct 7 — that they’d explicitly flagged as an un-forecastable long-tail series. The misses are macro-regime and tail calls; the hits are inertial series.
The AI under-call is the headline, and it’s worse than it looks because of when they forecast. The report published Feb 1 2023 — ChatGPT was two months old, GPT-4 six weeks away. They were staring directly at the inflection and still used a ~2.3×/yr compute trend (realized: ~4.7×/yr) and expected AI investment to plateau (realized: a roughly 6× run from $92B to ~$580B corporate in three years). Even a charitable reading — say their median priced a 2× investment rise by 2025 — still lands a full ~3–6× short. Pricing an exponential is hard even when it’s already underway.
Where the consensus was good, they were superb — and that’s most of the list. Population to within a rounding error, fertility, nuclear totals, energy, life expectancy, trust, same-sex-marriage share: all landed. These are the series where the UN/FAS/Energy-Institute projections are themselves strong, and the forecasters correctly didn’t fight them. Notably, the space-launch question — the one you pushed back on
nextbigfutureabout in that very thread — resolved at 4,510 in 2025, inside the pros’ 3,900–9,200 band; his 50k–500k was ~1–2 OOM high, and your “~1 OOM below his number” instinct bracketed the truth.The premise worth challenging
Your question was “how are the forecasts doing.” The sharper version is: at this horizon, the question barely discriminates. The thing that makes the Metaculus community distinctive — willingness to entertain exponentials, regime changes, and fat tails — is exactly what 1–3-year medians suppress, because over short windows the aggregate reverts to consensus. So a strong near-term scorecard (which this is) is mostly evidence that the consensus was right, and tells you almost nothing about whether the community’s tail-aware reasoning adds value. The one place that distinctive reasoning should have paid off — taking the AI exponential seriously — is precisely where they most under-used it.
Two implications:
The actual test of this tournament hasn’t run yet. Its entire design rationale was the 30/52/122-year horizons — where the OWID note explicitly says the community’s willingness to price black swans makes it less conservative than traditional sources. That bet is unfalsified until those resolve. Grading 2023/2025 is grading the warm-up.
For your AGI-timelines work specifically: the cleanest signal in this whole exercise is a base rate — sophisticated forecasters under-predict AI scaling even when watching it happen. Compute and capital both ran several-fold ahead of well-reasoned medians over three years. That’s a thumb on the scale toward the aggressive end of current AI forecasts, not the modal one.
Caveats on this grading
I’m scoring against the report’s prose descriptions, not the chart medians/IQRs — fine for the unambiguous calls, weaker for certifying “in range” on the hits. Three metrics (chickens, homicide, poverty) can’t be graded yet on their ~2-year-lagged sources. Two resolution problems are real and worth internalizing as a forecasting lesson in their own right: the genome-cost series simply stopped existing (NHGRI froze in 2022), and the “objects launched” and “nuclear” counts get revised upward for years after the fact, so any single-year resolution is a moving target. A forecast can be undone by the measurement pipeline rather than the world.
As an aside, I also asked the models to assess each other’s assessments above:
GPT-5.5 assesses Opus 4.8: “conceptual diagnosis is strong, but its scorecard is materially sloppier than mine on several important rows. The big difference is that it often grades the report’s prose instead of the report’s forecast table… it incorrectly treats fastest supercomputer and liberal democracy population as hits, and overstates the AI investment miss by mixing definitions”
I think the other model’s conceptual diagnosis is strong, but its scorecard is materially sloppier than mine on several important rows. The big difference is that it often grades the report’s prose instead of the report’s forecast table. That makes it more rhetorically satisfying, but less reliable as an audit.
Its best line is this: near-term OWID forecasts mostly test whether forecasters anchored well to institutional baselines, while the more distinctive Metaculus skill, pricing exponentials and weird tails, is barely visible at 1–3 years. I agree with that. It also correctly spots the AI compute undercall, the AI boom, the terrorism miss, the no-recession surprise, and the broader “inertial variables did well” pattern. Its own caveat says it is scoring against prose descriptions rather than exact medians/IQRs, and that caveat matters a lot.
Where I disagree: its tally is too generous. It says roughly 12 clean hits, 1 split, 4 misses, 1 measurement failure, 3 lagged-not-gradable. I would revise that downward, mainly because it incorrectly treats fastest supercomputer and liberal democracy population as hits, and overstates the AI investment miss by mixing definitions.
Quick comparison
Issue
Other model
My view
Overall framing
Strong: “near-term mostly grades consensus.”
I agree. This is probably the most important meta-point.
Use of exact forecast values
Weak: often uses prose/ranges.
I’d privilege the report tables and resolution definitions.
AI training compute
Correctly says miss low.
Agree, and probably the most important technical miss.
AI investment
Says biggest miss under.
Too strong. Depends on whether resolution uses narrow external private investment or broad corporate AI spending.
Supercomputer FLOPS
Calls it a hit.
I think this is wrong. Forecast was ~4.1–4.5e18 for 2025; actual TOP500 was 1.809e18.
Liberal democracies
Calls it directionally a hit.
I think this is wrong. Forecast was ~1.0–1.1B for 2025; V-Dem says 0.6B at end-2025.
Terrorism
Correctly says miss.
Agree, but it underplays the magnitude. Forecast was 19–20k; actual 2025 was 5,582.
CO₂
Says number hit, peak narrative premature.
Agree. That is a nice distinction.
Space objects
Calls hit.
Agree. Pros were impressively close.
Demographics
Calls population/fertility hits.
Mostly agree, though its fertility actuals are a little messy depending on UN/OWID projection vintage.
The most important factual disagreements
1. Supercomputer FLOPS: the other model is wrong to call this a hit
The report table gives the 2025 fastest-supercomputer forecasts as 4.1 × 10¹⁸ FLOP/s for Pros and 4.5 × 10¹⁸ for the public. The TOP500 November 2025 result was El Capitan at 1.809 exaFLOP/s. That is not a hit. It is a miss high by about 2.3–2.5×. The other model appears to have inferred from the prose that forecasters expected “sub-2 EF” by 2025, but the table does not say that. It says ~4 EF.
This is not a trivial bookkeeping dispute. It affects the 2032 adjustment. If you call 2025 a hit, you might leave the 2032 TOP500 forecast alone. If you call it a miss high, you should lower 2032, which is what I did.
2. AI investment: the other model overclaims the miss
The report table gives 2025 AI investment forecasts of $340B Pro and $489B public. The other model says actuals were “$581.7B corporate / $344.7B private” and therefore calls this the biggest miss under. But OWID’s relevant AI-investment chart says it focuses on external private-market investment and excludes internal corporate R&D, capex, public-sector funding, and public companies. On that narrower definition, $344.7B private makes the Pro forecast of $340B look almost exactly right, while the public forecast of $489B looks high. On broad corporate AI investment, the public forecast may be low.
So the better judgment is: direction right, metric definition decisive. The other model’s “biggest miss under” conclusion is only defensible if you resolve against broad corporate AI investment, not against the narrower OWID-style private-investment series.
3. Liberal democracies: the other model misses the U.S./V-Dem discontinuity
The report forecasted 1.00B Pro and 1.10B public people living in liberal democracies in 2025. V-Dem’s 2026 Democracy Report says only 7% of the world population, about 0.6B people, lived in liberal democracies at the end of 2025, and explicitly says the U.S. lost its long-term liberal-democracy status for the first time in over 50 years.
The other model calls this a directional hit, saying V-Dem shows a slow decline around ~1.0B. That looks stale or wrong. This is one of the biggest substantive misses in the whole set, because the forecast’s own resolution criteria used OWID or an alternative credible source for people living in liberal democracies, and the report itself flagged U.S. 2024 turmoil as a possible reason the U.S. might lose liberal-democracy status.
4. Terrorism: it gets the sign right but underplays the magnitude
The report forecasted 19k–20k terrorism fatalities for both 2023 and 2025. The Global Terrorism Index 2026 says 2025 deaths were 5,582, down 28% from 2024. It also says 2023 deaths spiked to 8,654, largely due to October 7, before falling again. So yes, the forecasters missed, but the other model frames the miss around “2023 rose 22%” instead of the more important fact that the median forecast was about 2.2× too high in 2023 and about 3.4× too high in 2025.
The right lesson is subtler than “black swan happened.” The October 7 shock pushed deaths upward, but even with that shock, the forecast was still far too high. The forecasters seem to have over-anchored to older high-terrorism baselines or to right-tail concerns.
5. CO₂: I like its distinction
Here I think the other model improves on my initial phrasing. The report’s Pro forecasts were 37.7 Gt for 2023 and 38.1 Gt for 2025, while the public forecasts were 36.4 and 36.2. The IEA says 2025 energy-related CO₂ reached a new high of nearly 38.4 Gt. So the Pro number was very good, the public number was too low, and the qualitative “at or near peak” narrative looks premature.
That “number right, story too early” framing is exactly the right way to score it.
Where the other model is better than mine
It is better than my first answer in three ways.
First, it has a cleaner theory of the errors: inertial variables good, regime/tail variables bad. That’s a useful compression. Population, space objects, fertility, energy, nuclear stockpiles, CO₂ Pro forecasts, and same-sex marriage mostly follow observable short-run trends. AI compute, AI investment, terrorism, GDP surprises, and liberal-democracy classification are more thresholdy or regime-sensitive.
Second, it correctly emphasizes that a 1–3-year scorecard is a weak test of the tournament’s real ambition. The tournament’s rationale was about 10-, 30-, and 100-year uncertainty. In the short run, even excellent forecasters often just become careful aggregators of UN, IMF, FAS, OECD, IEA, TOP500, OWID, and similar baselines. That is valuable, but it is not the same as proving far-future forecasting skill.
Third, it is right to treat AI as the most informative miss. The report’s 2025 AI training-compute forecasts were 2.9e25 Pro and 6.6e25 public, while Epoch now puts the largest known training run at around 5e26 FLOP, and notes that more than 30 models had crossed the 1e25 FLOP scale by June 2025.
Where mine is better
Mine is more careful about resolution definitions. That matters because these are not abstract “what happened in the world?” questions. They are “what number will a specified dataset or credible replacement source report?” questions. The other model sometimes slides from the metric to a nearby intuitive proxy: corporate AI spend instead of external private investment; prose expectations instead of table medians; total nuclear inventory instead of military stockpile-style series; broad democratic decline instead of the specific V-Dem/OWID liberal-democracy population count.
Mine is also less willing to count “directionally on track” as a hit when the forecast was numerical and the relevant year is missing or the actual value is far from the median. The other model’s “hit” threshold is too forgiving. A forecast of 1.0B liberal-democracy population resolving at 0.6B is not a hit just because the direction was “slightly down.” A forecast of 4.1e18 FLOP/s resolving at 1.809e18 is not a hit just because the prose said the leap might be delayed.
Revised synthesis after comparing both
I’d now summarize the 2023/2025 track record like this:
Cluster
My revised judgment
Very good
Space objects, world population, CO₂ Pro forecast, fertility, per-capita energy, nuclear stockpiles roughly, same-sex marriage roughly
Mixed or definition-sensitive
AI investment, genome sequencing, GDP per capita growth, OECD trust, G7 life expectancy, U.S. productivity
Clear misses
AI training compute low, terrorism high, liberal-democracy population high, fastest-supercomputer FLOPS high, public CO₂ too low
Not properly scoreable yet
Chickens, homicide, poverty, and several non-Pro/public-only OWID series with lagged data
So I’d move my earlier “B/B-minus overall” slightly downward for the fully scoreable subset, maybe closer to B-minus or C-plus if you grade medians strictly, while preserving B-plus-ish for slow-moving macro/demographic variables. The other model’s “12 clean hits” is too kind because two of those clean hits are not clean hits.
The best combined takeaway is this: Metaculus was good where disciplined base-rate anchoring was enough, but not clearly better than careful institutional extrapolation at 1–3 years. It underpriced AI scale-up, overestimated terrorism, missed a threshold political-classification shock, and overestimated TOP500 growth. For 2032, the right way to beat the original forecasts is to mostly leave the boring variables alone and make concentrated adjustments on the variables where these short-run misses reveal a wrong model class.
Opus 4.8 assesses GPT-5.5: “strong assessment — better than mine in several specific places because it has the actual Pro/Public medians I lacked — and corrects a real error I made. But also one significant flaw that fooled it, for the terrorism verdict (and, in a different way, fooled me too), and one overstatement, for supercomputer FLOPS”
This is a strong assessment, and in a few places it’s better than mine — including one spot where it cleanly corrects a real error I made. Before I adjudicate the places where we disagree (terrorism, supercomputers, liberal democracies), let me verify the two load-bearing facts behind its sharpest claims, since the whole comparison turns on who’s actually right.
V-Dem confirms the model: just 7% of the world (about 600 million people) lived in liberal democracies at the end of 2025 — the lowest in over 50 years — with the drop driven largely by the USA losing liberal-democracy status. So the model caught a real miss I was too soft on. Now the terrorism series, where I suspect the model’s “2.3× miss” may be a measurement artifact.
Both fact-checks came back clean, and they’re decisive. Verdict up front: this is a strong assessment — better than mine in several specific places because it has the actual Pro/Public medians I lacked — and it corrects a real error I made. But it also contains one significant flaw that fooled it (and, in a different way, fooled me too), and one overstatement. Combining its numbers with my provenance-checking beats either of us alone.
A reassuring meta-point first: the medians it reports are almost certainly authentic, not hallucinated. The tell is internal consistency with the underlying data series — e.g. its terrorism median (~20k) matches the GTD baseline the question actually resolves on, and its AI-investment Pro median ($340B) matches the narrow-private actual almost exactly. A model fabricating numbers wouldn’t reliably reproduce series-specific baselines it had no other reason to know. That said, the entire numerical scorecard rests on those transcriptions, so they’re the one thing worth spot-checking against the report PDF.
Where it’s right and corrects me
AI investment — it catches a genuine error of mine, and this is the biggest single correction. In turn 1 I called this “the biggest under-forecast” by comparing the forecast to ~$582B of broad corporate AI investment. That was the wrong denominator. The OWID metric is narrow external private funding for privately-held AI companies, which excludes Big Tech capex, public firms, and public-sector money — and on that series the 2025 actual is ~$345B. So the Pro median of $340B was nearly exact, and the Public’s $489B was ~40% high. The model’s framing — “directionally right, hard to score, looks high on narrow private funding” — is correct and mine was wrong. And it’s a more interesting result than my version: forecasting $340B in Feb 2023, off a falling $92B base, means the Pros actually did price the investment boom on this metric. I had the story backwards.
Liberal democracies — it’s right, I was too lenient. I waved this through as a “directional hit (noisy metric).” It’s a clear miss high: forecast ~1.0B, actual ~0.6B (7% of the world) at end-2025, the lowest in 50+ years, largely because the USA lost its liberal-democracy status — the first time in more than 50 years, falling to electoral democracy. The model names the mechanism precisely (“overexpected institutional continuity in democracy”). I undersold a real miss.
Nuclear — cleaner than mine. Its medians (~9.4–9.5k) make clear the question resolves on the military stockpile (~9,600), which is exactly right and resolves the total-vs-stockpile ambiguity I left dangling in turn 1.
Where I’d push back on it (and where I was also wrong)
Terrorism — its sharpest-looking verdict is its biggest flaw, and my turn-1 take was wrong too. The model calls this a “bad miss high… overestimated by ~2.3× in 2023 and ~3.4× in 2025.” But it compared the forecast to the wrong database. The OWID metric resolves on the Global Terrorism Database (GTD), on which ~20,000 people died from terrorism in 2019 — so the Pro median of ~19–20k isn’t an overestimate at all; it’s a sensible extrapolation of the actual resolution series. Two problems then compound:
The GTD is discontinued. As of 2025 the GTD is closed, and the data only extend through 2020/2021. There is no native 2023 or 2025 value — this question is unresolvable, exactly like the genome-cost case (NHGRI frozen at 2022). It belongs in the not-scoreable bucket, not the miss column.
The “actual” it used is a different, lower-counting database. The 8,352 / 5,582 figures are from the GTI, which systematically counts fewer deaths than the GTD — GTI put 2019 at 13,826 versus GTD’s ~20,000 for the same year. So a chunk of the apparent “2.3× miss” is just the GTD→GTI level shift, not forecaster error.
And the kicker: on the GTI proxy, deaths actually fell to 5,582 in 2025, the lowest since 2007 — so the forecasters’ qualitative “continue to decline” call was directionally vindicated by 2025. My own turn-1 framing (“predicted decline, got an Oct-7 spike to 8,352, miss”) was also wrong: I used the GTI number, missed that the GTD is the dead resolution series, and treated a one-year blip as the verdict. We both mis-scored this in opposite directions; the correct call is “unresolvable / not cleanly scoreable, decline directionally right.”
Supercomputer FLOPS — “clear miss high” overstates it. Forecast 4.1e18 vs actual 1.809e18 is 2.3× — but that’s 0.36 of an order of magnitude on a log-scale exponential metric. On the same log-error basis the model (correctly) uses to call AI-compute a miss (~0.9 OOM low), a 0.36-OOM overshoot is a mild overshoot, not a “clear miss.” The directional read (“expected the exascale race to accelerate faster than the TOP500 benchmark did”) is right; the severity is dialed too high. There’s also a metric-decoupling issue: the frontier AI clusters (xAI, Meta) don’t run the LINPACK benchmark, so TOP500′s #1 increasingly fails to measure the thing that matters.
Where we agree
The structural/slow-moving hits (population, fertility, CO₂ level, energy, space objects, same-sex marriage); AI training compute as a genuine miss-low (its “~5e26 Grok 4 vs 6.6e25 Public median” and my “~1e26+ vs an extrapolated median” both land at ~1 OOM under); and the laggy third that simply isn’t scoreable yet (chickens, poverty, homicide, genome cost, productivity, GDP growth). On that last bucket the model is actually better organized than I was — its dedicated table with specific reasons (PPP-vintage shifts, NHGRI’s 2022 freeze, Penn World Table ending at 2023) is cleaner than my prose.
The synthesis that beats both assessments
Here’s the reconciled picture of the contested calls:
Metric
Model’s verdict
My turn-1 verdict
Reconciled (what’s actually true)
AI investment
Directionally right, looks high (narrow series)
“Biggest under-forecast”
Model right. Pro $340B ≈ exact vs ~$345B; I used the wrong denominator
Liberal democracies
Clear miss high (US reclassified)
“Directional hit, noisy”
Model right. ~1.0B forecast vs 0.6B actual; I was too lenient
Terrorism
Bad miss high (~2.3×)
Miss (Oct-7 spike)
Both wrong. GTD discontinued + GTI mismatch → unresolvable; decline directionally right
Supercomputer FLOPS
Clear miss high
Hit
Between us. Mild 0.36-OOM overshoot + metric decoupling
Nuclear
Very good
Hit (waffled on series)
Model cleaner (military stockpile, ~9.6k)
AI compute
Miss low (~1 OOM)
Miss low
Agree
Lay these side by side and a pattern jumps out that neither assessment states: of the model’s four headline “misses,” only AI compute is a clean real-world forecasting error. Terrorism is a dead data source plus a database mismatch; liberal democracies is a V-Dem reclassification of the US (a classification event, unforeseeable in Feb 2023, not a smooth trend the forecasters misread); supercomputers is a 0.36-OOM overshoot on a metric that’s decoupling from reality. Three of the four are measurement/definition artifacts, not failures to read the world.
That has two consequences. First, it means the forecasters look better than the model’s B/B-minus — closer to B-plus/A-minus on real-world calibration, with the honest caveat that roughly a third of the metrics are measurement-fragile. The model’s grade is slightly too harsh because it scored three measurement artifacts as forecasting misses. Second, it sharpens my turn-1 thesis: I argued the 1–3yr exercise mostly grades the consensus they anchored on; the better statement now is that the dominant failure mode at this horizon isn’t bad forecasting, it’s metric fragility — series get discontinued (GTD, NHGRI), redefined (V-Dem reclassification), or measure the wrong proxy (TOP500 vs AI clusters). The one place the world itself outran them is AI compute. Everything else that “missed” is the ruler changing, not the thing being measured.
Net: the model’s assessment is genuinely good and I’d trust its numbers over my turn-1 qualitative grading wherever they conflict — it fixed my AI-investment and liberal-democracy errors outright. What I’d add to it is provenance discipline (which rescues terrorism from the miss column and softens supercomputers) and the measurement-artifact reframe. The combined verdict is more accurate than either of us produced alone, which is the whole point of running two passes.
I’ll let Opus 4.8 have the final word:
… of the model’s four headline “misses,” only AI compute is a clean real-world forecasting error. Terrorism is a dead data source plus a database mismatch; liberal democracies is a V-Dem reclassification of the US (a classification event, unforeseeable in Feb 2023, not a smooth trend the forecasters misread); supercomputers is a 0.36-OOM overshoot on a metric that’s decoupling from reality. Three of the four are measurement/definition artifacts, not failures to read the world. … the dominant failure mode at [the 1-3 year] horizon isn’t bad forecasting, it’s metric fragility — series get discontinued (GTD, NHGRI), redefined (V-Dem reclassification), or measure the wrong proxy (TOP500 vs AI clusters). The one place the world itself outran them is AI compute. Everything else that “missed” is the ruler changing.
Yeah, even smaller sums can still be trajectory-changing, cf. Nadia Asparouhova’s Helium Grants ($1.5k average):
(screenshots because the text isn’t copy-paste-able)
Nadia also has advice on starting a microgrants program (including plenty of logistics advice, to your point re: optimising to reduce admin costs) and a list of other microgrant programs on GitHub.
Speaking mostly to myself—I was reminded of this post when I saw Nabeel Qureshi’s recent Substack note
Multiple Claude-written articles with 10k+ likes here recently. People seem to really like slop on X too. Reluctant conclusion: maybe the taste for organic hand rolled human only tokens will become a niche IQ taste similar to enjoying Japanese wood carvings or something.
and Evan Armstrong’s reply (see also his In Defense of AI Slop, h/t Gwern)
Correct—I scrapped Substack using Pangrams API and found that readers just can’t really tell if it is AI slop or not. It’s the beginning of the end for writers who purely compete on the basis of prose.
re: “When do we expect LLMs to become capable of writing online content that we actually think is worth reading?[6]” I still don’t think they’re there yet for me personally, nor many LWers probably (maybe Mythos-class models given their own Substacks like Opus 3 might change my mind; eudaemon_0 is certainly intriguing), but it’s hard to deny that they’ve crossed the bar for a lot of people online, and it’s pretty annoying.
Disproving the Dinitz-Garg-Goemans conjecture with GPT 5.6 Pro, a story in four acts:
In contrast to Kerger’s 10-page effortprompt, Dmitry Rybin basically just said “do a breakthrough”.