Thoughts on whether the >10x lower chip-to-chip interconnect from the CPX chips (PCIe 6.0x16′s 128GB/s unidirectional vs. NVLink 5′s 1.8TB/s bidirectional) will be a bottleneck blocking them from being that useful in pre-training?
romeo
Karma: 865
A 2032 Takeoff Story
CloudMatrix announcements indeed predated AI 2027 but the compute forecast did make predictions of how much compute China will have, including domestic production, smuggling and legal purchasing of foreign chips and found that they would still be significantly behind by 2027. The CloudMatrix doesn’t change this because its still around 2x less cost-efficient than what US companies have access to, and US companies are investing around 4-5x their Chinese counterparts. This follow up blog post addressed the concern that we underestimated China, focusing on this compute gap.
I think China has a very serious chance of overtaking the US in terms of both compute and overall frontier AI capabilities post-2030, since they might crack EUV by then and the US will start running into more significant power bottlenecks that China won’t face.
How 2025 AI Forecasts Fared So Far
I agree that its very plausible that China would steal the weights of Agent-3 or Agent-4 after stealing Agent-2. This was a toss up when writing the story, we ultimately went with just stealing Agent-2 for a combination of reasons. From memory the most compelling were something like:
OpenBrain + the national security state can pretty quickly/cheaply significantly increase the difficulty and importantly the lead time required for another weights theft.
Through 2027 China is only 2-3 months behind, so the upside from another weights theft (especially when you consider the lead time needed) is not worth the significantly increased cost from (1).
Something that is a bit more under-explored is the potential sabotage between the projects. We have the uncertain assumption that the efforts on both sides would roughly cancel out, but we were quite uncertain on the offense-defense balance. I think a more offense favored reality could change the story quite a bit, basically with the idea of MAIM slowing both sides down a bunch.
FYI Scott Alexander wrote up AI 2027: Media, Reactions, Criticism
so maybe the second 2,000x of scaling should be reached by 2045 instead.
Yeah sounds reasonable, that would match up with my 1.56x/year number, so to summarize, we both think this is roughly plausible for 2028-2045?
1.3x/year (compute production) x 1.2x/year (compute efficiency) ~= 1.55x/year (compute available)1.1x/year (investment) x 1.4x/year (price performance) ~= 1.55x/year (compute available)
So a 3x slowdown compared to the 2022-2028 trend (~3.5x/year).
To be clear I don’t think the profit margin is the only thing that explains the discrepancy.
I think the relevant question is more like: under my method, is 1.3x (production) x 1.2x (hardware) = 1.56x/year realistic over 2-3 decades or am I being too bullish? You could ask an analagous thing about your method (i.e., is 1x investment and 1.4x price performance realistic over the next 2-3 decades?) Two different ways of looking at it that should converge.
If i’m not being too bullish with my numbers (which is very plausible, e.g., it could easily be 1.2x, 1.2x), then i’d guess the discrepancy with your method comes from it being muddled with economic factors (not just chip designer profit margins but supply/demand factors affecting costs across the entire supply chain, e.g., down to like how much random equipment costs and salaries for engineers). Maybe 1x investment is too low, maybe should be multiplied with inflation and GDP growth?
When funding stops increasing, the current pace of 3.55x per year (fueled by increasing funding) regresses to the pace of improvement in price-performance of compute of 1.4x per year, which is 3.7x slower. If the $140bn training systems of 2028 do get built, they’ll each produce about 1.5e22 BF16 FLOP/s of compute, enough to train models for about 5e28 BF16 FLOPs.
This is a nice way to break it down, but I think it might have weird dependencies e.g., chip designer profit margins.
Instead of:training run investment ($) x hardware price performance (FLOP/$) = training compute (FLOP)
Another possible breakdown is:hardware efficiency per unit area (FLOP/s/mm^2) x global chip production (mm^2) x global share of chips used in training run (%) * training time (s) = training compute (FLOP)
This gets directly at the supply side of compute. It’s basically ‘moore’s law x AI chip production x share of chips used’. In my model for the next three years are 1.35x1.65x1.5 ~= 3.4x, so matches your 3.55x/year pretty closely. Where we differ slightly I think would be in the later compute slowdown.
Under my model there are also one-time gains happening in AI chip production and share of chips used (as a result of the one-time spending gains in your model). Chip production has one-time gains because AI only uses 5-10% of TSMC leading nodes and is using up spare capacity as fast as packaging/memory can be produced. Once this caps out, I think the 1.65x will default to being something like 1.2-4x as it gets bottlenecked on fab expansion (assuming like you said an investment slowdown). ‘Share of chips used’ growth goes to 1x by definition.
Even taking the lower end of the estimate would mean that ‘moore’s law’ hardware gains would have to slow down ~2x to 1.16x/year to match your 1.4x number. I do think hardware gains will slow somewhat but 1.16x is below what I would bet. Taking my actual medians, I think i’m at like 1.3x (production) x 1.2x (hardware) = 1.56x/year so more like a 2.8x slowdown, not 3.7x slowdown.
So resolving the discrepancy, it seems like my model is basically saying that your model overestimates the slowdown because it assumes profit margins stay fixed, but instead under slowing investment growth these should collapse? That feels like it doesn’t fully explain it though since it seems like it should be a one time fall (albeit a big one). Maybe in the longer term (i.e. post-2035) I agree with you more and my 1.3x production number is too bullish.
A brief history of things that have defined my timelines to AGI since learning about AI safety <2 years ago
Bio anchors gave me a rough ceiling around 1e40 FLOP for how much compute will easily make AGI.
Fun with +12 OOMs of Compute brought that same ‘training-compute-FLOP needed for AGI’ down a bunch to around 1e35 FLOP.
Researching how much compute is scaling in the near future.
At this point I think it was pretty concentrated across ~1e27 − 1e33 flop so very long tail and something like a 2030-2040 50% CI.
The benchmarks+gaps argument to partial AI research automation.
The takeoff forecast for how partial AI research automation will translate to algorithmic progress.
The trend in METR’s time horizon data.
At this point my middle 50% CI is like 2027 − 2035, and would be tighter if not for a long tail that I keep around just because I think it’s have a bunch of uncertainty. Though I do wish I had more arguments in place to justify the tail or make it bigger, ones that compete in how compelling they feel to me to the ones above.
Thanks for linking. I skimmed the early part of this post because you labelled it explicitly as viewpoints. Then I see that you engaged with a bunch of arguments about short timelines, but they are all pretty weak/old ones that I never found very convincing (the only exception is that bio anchors gave me an early ceiling early on around 1e40 FLOP for compute needed to make AGI). Then you got to LLMs and acknowledged:
The existence of today’s LLMs is scary and should somewhat shorten people’s expectations about when AGI comes.
But then gave a bunch of points about the things LLMs are missing and suck at, which I already agree with.
Aside: Have I mischaracterized so far? Please let me know if so.
So, do you think you have arguments against the ‘benchmarks+gaps argument’ for timelines to AI research automation, or why AI research automation won’t translate to much algorithmic progress? Or any of the other things that I listed as ones that moved my timelines down:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch to around 1e35 FLOP
Researching how much compute is scaling.
The benchmarks+gaps argument to partial AI research automation
The takeoff forecast for how partial AI research automation will translate to algorithmic progress.
The recent trend in METR’s time horizon data.
But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.
Totally agree! Thank you for phrasing it elegantly. This is basically what I commented on Ege’s post yesterday, I asked him to engage with the actual crux and make arguments about why the software-only singularity is unlikely.
There’s an entire class of problem within ML that I would see as framing problems and the one thing I think LLMs don’t help that much with is framing.
Could you say more about this? What do you mean by framing in this context?
There’s this quote I’ve been seeing from Situation Awareness that all you have to do is “believe in a straight line on a curve” and when I hear that and see the general trend extrapolations my spider senses start tingling.
Yeah that’s not really compelling to me either. SitA didn’t move my timelines. Curious if you have engaged with the benchmarks+gaps argument to AI R&D automation (timelines forecast), and then the AI algorithmic progress that would drive (takeoff forecast). These are the things that actually moved my view.
Thanks for the link, that’s compelling.
By far the most top-voted answer was one arguing that we might get AI to substantially accelerate AI progress because a particular AI research engineering benchmark looks like it will get saturated within a couple of years.
The list of things I see as concrete arguments that have moved down my timelines include exactly this!
But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability.
...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each.
The benchmarks also do not take into account the fact that the vast majority of them measure a model’s performance in a situation where the model is only given one task at a time, and it can completely focus on solving that...
Agree that there are many such ‘gaps’! Would be curious to hear if you think there are important ones missing from the timelines forecast or if you have strong views that some of them will be importantly harder!
Furthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don’t meet the spirit of it.
Thanks for this! This is a good point. Do you think you can go further and say why you think it will be very hard to fix in the near term, so much so that models won’t be useful for AI research?
I agree re Leopold’s piece, it didn’t move my timelines.
A brief history of the things that have most collapsed my timelines down since becoming aware of AI safety <2 years ago:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch
Generally working on AI 2027, which has included
Writing and reading the capabilities progression where each step seems plausible.
Researching how much compute is scaling.
Thinking about how naive and limiting current algorithms and architectures seem, and what changes they are plausibly going to be able to implement soon.
The detailed benchmarks+gaps argument in the timelines forecast.
The recent trend in METR’s time horizon data.
romeo’s Shortform
Maybe i’m in an echo chamber or have just had my head in the sand while working on AI 2027, but now that i’ve been paying attention to AI safety for almost 2 years and seen my timelines gradually collapse, I really want to engage with compelling arguments that might lengthen my timelines again.
I have feel like there are a bunch of viewpoints expressed about long timelines/slow takeoff but a lack of arguments. This is me reaching out in the hope that people might point me to the best existing write-ups or maybe make new ones!
I am tracking things like: “takeoff will be slow because of experiment compute bottlenecks,” or “timelines to AIs with good research taste are very long,” or even more general “look how bad AI is at all this (not-super-relevant-to-a-software-only-singularity-)stuff that is so easy for humans!” but in my opinion, these are just viewpoints (which by the way, seem to often get stated very confidently in a way that makes me not trust the epistemology behind them). So sadly these statements don’t tend to lengthen my timelines.
In my view, these viewpoints would become arguments if they were more like (excuse the spitballing):“1e28 FLOPs of experiment compute is unlikely to produce much algorithmic progress + give a breakdown of why a compelling allocation of 1e28 FLOP doesn’t get very far”
“Research taste is in a different reference class to the things AI has been making progress on recently + compelling reasoning, like, maybe:
‘it has O(X) more degrees of freedom,’
‘it has way less existing data and/or it’s way harder to create data, or give a reward signal’
‘things are looking grim for the likelihood of generalization to these kinds of skills’
“there are XYZ properties intelligence needed that can’t be simulated by current hardware paradigms”
Currently I feel like I have a heavy tail on my timelines and takeoff speeds as a placeholder in lieu of arguments like this, that i’m hoping exist.
Thanks for the detailed comments! We really appreciate it. Regarding revenue, here’s some thoughts:
“AI 2027” forecasts that OpenAI’s revenue will reach 140 billion in 2027. This considerably exceeds even OpenAI’s own forecast, which surpasses 125 billion in revenue in 2029. I believe that the AI 2027 forecast is implausible.[4]
AI 2027 is not a median forecast but a modal forecast, so a plausible story for the faster side of the capability progression expected by the team. If you condition on the capability progression in the scenario, I actually think $140B in 2027 is potentially on the conservative side. My favourite parts of the FutureSearch report is the examples from the ~$100B/year reference class, e.g., ‘Microsoft’s Productivity and Business Process segment.’ If you take the AI’s agentic capabilities and reliability from the scenario seriously, I think it feels intuitively easy to imagine how a similar scale business booms relatively quickly, and i’m glad that FutureSearch was able to give a breakdown as an example of how that could look.
So maybe I should just ask whether you are conditioning on the capabilities progression or not with this disagreement? Do you think $140b in 2027 is implausible even if you condition on the AI 2027 capability progression?
If you just think $140B in 2027 is not a good unconditional median forecast all things considered, then I think we all agree!Note: “AI 2027” chooses to call the leading lab “OpenBrain”, but FutureSearch is explicit that they’re talking about OpenAI.
We aren’t forecasting OpenAI revenue but OpenBrain revenue which is different because its ~MAX(OpenAI, Anthropic, GDM (AI-only), xAI, etc.).[1] In some places FutureSearch indeed seems to have given the ‘plausible $100B ARR breakdown’ under the assumption that OpenAI is the leading company in 2027, but that doesn’t mean the two are supposed to be equal neither in their own revenue forecast nor in any of the AI 2027 work.
FutureSearch’s estimate of paid subscribers for April 2025 was 27 million; the actual figure is 20 million. They justify high expected consumer growth with reference to the month-on-month increase in unpaid users from December 2024 → February 2025. Data from Semrush replicates that increase, but also shows that traffic has since declined rather than continuing to increase.
The exact breakdown FutureSearch use seems relatively unimportant compared to the high level argument that the headline (1) $/month and (2) no. of subscribers, very plausibly reaches the $100B ARR range, given the expected quality of agents that they will be able to offer.
Looking at market size estimates, FutureSearch seems to implicitly assume that OpenAI will achieve a near-monopoly on Agents, the same way they have for Consumer subscriptions. Enterprise sales are significantly different from consumer signups, and OpenAI doesn’t currently have a significant technical advantage.
I don’t think a monopoly is necessary, there’s a significant OpenBrain lead-time in the scenario, and I think it seems plausible that OpenBrain would convert that into a significant market share.
- ^
Not exactly equal since maybe the leading company in AI capabilities (measured by AI R&D prog. multiplier), i.e., OpenBrain, is not the one making the most revenue.
Also see FutureSearch’s report on a plausible breakdown for how OpenBrain hits $100B ARR by mid-2027.
I think if you condition on the capability progression in the scenario and look at existing subscription services generating in the $100B range, it feels very plausible intuitively, independently from the ‘tripling’ extrapolation.
Do you have a take on ‘all things considered p(doom) / p(bad outcome)’ by plan? My guess is the EV from moving to plan A doesn’t look great on pure takeover risk but looks better on the all things considered risk.
In terms of how likely each regime is to happen, I feel like Plan C is modal, not Plan D and Plan B is already similarly likely to Plan D. My A / B / C / D / E is probably 5 / 25 / 35 / 30 / 5. Main reasons is that I don’t see current lab leaders actually wanting to do D, seems like they probably all will want to do C to me when the time comes, enough so to probably just do it even if there’s a chance of losing race.
In terms of all things considered p(doom) my A / B / C / D / E is probably 12 / 35 / 40 / 70 / 85.
So key takes that i feel are:
- Plan B doesn’t feel that much better than Plan C to me.
- Plan C is probably already the default.
- A lot of people are working on increasing Plan B mass which it seems not high EV to me.
- Most tractable highest EV things to work on IMO seem like increasing Plan A mass, and reducing plan C p(doom).