Also, this might be a place where people who are interested in learning more about how they could get involved in working on AI verification come and look, so i’m plugging the ‘get involved in verification’ page of Plan A here: https://ai-2040.com/verification. In particular the EOI form, where I share your response with relevant verification funders / companies for them to reach out to you about opportunities.
romeo
Karma: 1,550
Yeah these are promising signs, but I still think this deserves to be an Open Question going forwards in case it changes? Seems quite plausible to me that the overhead would be higher in the future (maybe even currently) on OpenAI / Anthropic production inference.
Thank you for writing this, it’s so great! I’ll respond line by line below:
OQ2: How can interconnect limits allow for video generation, which produces a lot of egress?
I agree that we care more about egress bandwidth (it seems like the bottleneck on using the monitored compute for training) but we might also care about a lot the ingress bandwidth, because the 1000x smaller dark compute could be used to do training while the monitored compute is being used for massive RL rollouts / data generation, and then using ingress video bandwidth (that we might want to allow to be very high for e.g., computer using agents or future robot-controlling-models) to upload new weights / low rank updates super quickly. So maybe additional open questions should be “how can interconnect limits allow for video inputs, which produce a lot of ingress?” This interacts with the memory wiping stuff below.
But I don’t understand this: why does it matter what a rack has in memory, provided every bit it emits is verified?
It’s mostly as defense in depth against spare compute being used for unverified stuff communicated out via side channels / covert channels (or tiny slippage in the verification to the extent we can’t get that to be literally 0). On this note, i think something like “How low can we get side channels bandwidth to be?” deserves to be its own Open Question.
Unfortunately, in an initial test, Amodo Design found that provably wiping a rack took ~24h and, if I’m reading their results correctly, left more than 100TB of memory unwiped.
A baseline proposal I was imagining (that I might not have included in the final writeup? oops if so) is that you have SSDs only allowed outside of the ‘inference-units’ (i.e., outside of the racks) so you avoid the unwiped 100TB, and then after memory wipes, you allow fast uploads from a bank of whitelisted model weights. If you did this alone I think you’d get to the 43 minutes according to their post? Then maybe you could do a hot reboot of the power going to the rack (to avoid the cold boot attacks that might persist volatile memory) instead of forced memorization and I would guess you could do even better? I’m surprised by the 24h latency though in their forced-memorization proposal, will look into it more.
Maybe another proposal would be to repurpose the backend networking to connect racks to a few ‘memory verifier’ servers, that the verifier controls, and uses to randomly choose a rack to refresh a rack by uploading a bunch of random noise to and checking it returns it perfectly. With 4 TB/s channels i’d guess this brings it down to ~minutes?Plan A says that replay is used to “check the correctness of the packets”. I find this phrasing confusing: how are the packets produced by training any less “correct” than those produced by inference? My understanding is that what replay buys us depends on whether the replay server hardware is trusted by the verifier or by the prover (I assume that manufacturing a mutually-trusted replay server is impractical). I sketch the sorts of properties one could verify in each case below.
Two things here. (1) part of the proposal is for there to be a set of whitelisted inference-code + whitelisted set of model weights P*, so it’s easy to check that P \in P*. Sorry this wasn’t clear. I think this resolves OQ4?
(2) Yes, I think we only need it to be unilaterally trusted by the verifier.
Agree that some cruxes here include (a) overhead from the deterministic stack, and (b) what % of samples need to be replayed for high detection probability (which as you say seems small due to power of sampling). Out of these, i think (a) might deserve to be its own Open Question?The trust issues in this proposal seem quite bad to me: algorithmic secrets and sensitive user information are probably leaked.
Agree this is a potential problem (especially in versions of Plan A that are aiming for lower transparency than our default proposal, where this stuff is (mostly) supposed to be transparent anyway), so this also deserves to be an Open Question i think: “What maximum covert egress can we limit a verifier-trusted replay server to from the prover’s perspective?”
On the untrusted replay section, Daniel Reuter’s work indeed seems very promising here! Hopefully it works, I would be very excited to see the summary you said you might write up. I’m worried about how you would verify the “whitelisted model weights” part of the proposal though (i.e., to defend from a big RL / federated learning attack), but maybe we can just do that via the memory wipes + ingress/egress limits.On ZKPs: [edits as of Aug 5 ’26]
I think you are correct thatthe 1e6pretty high overhead might be fine because it benefits from the sampling math. This was a simple oversight on my side, because I didn’t connect until now that you can do the proofs post-hoc via challenges (i was previously naively imagining the prover needing to generate proofs for all tokens, rather than there being random challenges with e.g., 1/1e9 probability after the tokens are generated), thank you for surfacing this.Like you, i’m now also curious if anyone else can explain if this is missing something?
I talked with some ZKP experts, and the problem with the 1e6 overhead is that you need to generate the proof for each token in context, which adds several OOMs (e.g., 4 OOMs with avg. 10k context). Apparently there are forthcoming solutions to this though.
Some further potential complications / questions i have on ZKPs are:
- what is the required reproducibility in the ZKP case? e.g., is it much higher than the replay case because of the ZKP being more sensitive to tiny differences than the replay?
- how robust can we make the ZKPs? Apparently there’s been a ‘track record of under-constrained circuits’ that let provers prove false statements silently? E.g., https://arxiv.org/abs/2402.15293, https://blog.trailofbits.com/2022/09/15/it-pays-to-be-circomspect/
I’ve come back to register here that i’ve updated much closer to your views on likelihood but i’m (seemingly) more pessimistic on doom (unless your all things considered p(doom) is a bunch higher than takeover risk).
These numbers imply overall p(doom) around 55%
My understanding is the hardware performance trend in FLOP/area of 1.35x/yr (precision and sparsity constant) has mostly been driven by transistor density ~1.25x/yr (transistors/area) and little by Nvidia chip design ~1.1x/yr (flop/transistor). I don’t think its true that the transistor density (traditional ‘Moore’s law’) has stopped, it seems like its only slowed down at most around 2x. My best guess conditional on no AGI by 2045, is that we see a smooth continuation of gradually slower transistor density improvement in the traditional paradigms (i.e., ‘2nm’ branded and ‘1.4nm’ branded nodes that use ASML’s new High-NA, and planned ‘Hyper-NA’) that i’d guess will average around 1.1-1.2x/yr, and/or might see some relatively more spikey progress from some paradigm shifts in computing and chip fabrication. So my overall, all things considered guess for FLOP/area is that it’ll average around 1.25x/yr through 2045 | no AGI. Then to get to price performance trend (FLOP/$), you have to multiply by the area/$ trend. I think with no AGI by 2045, you probably get a 4x one-time gain from Nvidia’s margin gradually going down, and then a steady trend of general manufacturing and economic efficiency bringing down manufacturing costs, naively like 5%/yr. That nets out at around a combined 1.1x/yr trend in area/$, so combined with the 1.25x/yr trend in FLOP/area, that’s why i expect the hardware price performance trend to average around 1.4x/yr through 2045 conditional on no AGI.
Do you have a take on ‘all things considered p(doom) / p(bad outcome)’ by plan? My guess is the EV from moving to plan A doesn’t look great on pure takeover risk but looks better on the all things considered risk.
In terms of how likely each regime is to happen, I feel like Plan C is modal, not Plan D and Plan B is already similarly likely to Plan D. My A / B / C / D / E is probably 5 / 25 / 35 / 30 / 5. Main reasons is that I don’t see current lab leaders actually wanting to do D, seems like they probably all will want to do C to me when the time comes, enough so to probably just do it even if there’s a chance of losing race.
In terms of all things considered p(doom) my A / B / C / D / E is probably 12 / 35 / 40 / 70 / 85.
So key takes that i feel are:
- Plan B doesn’t feel that much better than Plan C to me.
- Plan C is probably already the default.
- A lot of people are working on increasing Plan B mass which it seems not high EV to me.
- Most tractable highest EV things to work on IMO seem like increasing Plan A mass, and reducing plan C p(doom).
Thoughts on whether the >10x lower chip-to-chip interconnect from the CPX chips (PCIe 6.0x16′s 128GB/s unidirectional vs. NVLink 5′s 1.8TB/s bidirectional) will be a bottleneck blocking them from being that useful in pre-training?
CloudMatrix announcements indeed predated AI 2027 but the compute forecast did make predictions of how much compute China will have, including domestic production, smuggling and legal purchasing of foreign chips and found that they would still be significantly behind by 2027. The CloudMatrix doesn’t change this because its still around 2x less cost-efficient than what US companies have access to, and US companies are investing around 4-5x their Chinese counterparts. This follow up blog post addressed the concern that we underestimated China, focusing on this compute gap.
I think China has a very serious chance of overtaking the US in terms of both compute and overall frontier AI capabilities post-2030, since they might crack EUV by then and the US will start running into more significant power bottlenecks that China won’t face.
I agree that its very plausible that China would steal the weights of Agent-3 or Agent-4 after stealing Agent-2. This was a toss up when writing the story, we ultimately went with just stealing Agent-2 for a combination of reasons. From memory the most compelling were something like:
OpenBrain + the national security state can pretty quickly/cheaply significantly increase the difficulty and importantly the lead time required for another weights theft.
Through 2027 China is only 2-3 months behind, so the upside from another weights theft (especially when you consider the lead time needed) is not worth the significantly increased cost from (1).
Something that is a bit more under-explored is the potential sabotage between the projects. We have the uncertain assumption that the efforts on both sides would roughly cancel out, but we were quite uncertain on the offense-defense balance. I think a more offense favored reality could change the story quite a bit, basically with the idea of MAIM slowing both sides down a bunch.
FYI Scott Alexander wrote up AI 2027: Media, Reactions, Criticism
so maybe the second 2,000x of scaling should be reached by 2045 instead.
Yeah sounds reasonable, that would match up with my 1.56x/year number, so to summarize, we both think this is roughly plausible for 2028-2045?
1.3x/year (compute production) x 1.2x/year (compute efficiency) ~= 1.55x/year (compute available)1.1x/year (investment) x 1.4x/year (price performance) ~= 1.55x/year (compute available)
So a 3x slowdown compared to the 2022-2028 trend (~3.5x/year).
To be clear I don’t think the profit margin is the only thing that explains the discrepancy.
I think the relevant question is more like: under my method, is 1.3x (production) x 1.2x (hardware) = 1.56x/year realistic over 2-3 decades or am I being too bullish? You could ask an analagous thing about your method (i.e., is 1x investment and 1.4x price performance realistic over the next 2-3 decades?) Two different ways of looking at it that should converge.
If i’m not being too bullish with my numbers (which is very plausible, e.g., it could easily be 1.2x, 1.2x), then i’d guess the discrepancy with your method comes from it being muddled with economic factors (not just chip designer profit margins but supply/demand factors affecting costs across the entire supply chain, e.g., down to like how much random equipment costs and salaries for engineers). Maybe 1x investment is too low, maybe should be multiplied with inflation and GDP growth?
When funding stops increasing, the current pace of 3.55x per year (fueled by increasing funding) regresses to the pace of improvement in price-performance of compute of 1.4x per year, which is 3.7x slower. If the $140bn training systems of 2028 do get built, they’ll each produce about 1.5e22 BF16 FLOP/s of compute, enough to train models for about 5e28 BF16 FLOPs.
This is a nice way to break it down, but I think it might have weird dependencies e.g., chip designer profit margins.
Instead of:training run investment ($) x hardware price performance (FLOP/$) = training compute (FLOP)
Another possible breakdown is:hardware efficiency per unit area (FLOP/s/mm^2) x global chip production (mm^2) x global share of chips used in training run (%) * training time (s) = training compute (FLOP)
This gets directly at the supply side of compute. It’s basically ‘moore’s law x AI chip production x share of chips used’. In my model for the next three years are 1.35x1.65x1.5 ~= 3.4x, so matches your 3.55x/year pretty closely. Where we differ slightly I think would be in the later compute slowdown.
Under my model there are also one-time gains happening in AI chip production and share of chips used (as a result of the one-time spending gains in your model). Chip production has one-time gains because AI only uses 5-10% of TSMC leading nodes and is using up spare capacity as fast as packaging/memory can be produced. Once this caps out, I think the 1.65x will default to being something like 1.2-4x as it gets bottlenecked on fab expansion (assuming like you said an investment slowdown). ‘Share of chips used’ growth goes to 1x by definition.
Even taking the lower end of the estimate would mean that ‘moore’s law’ hardware gains would have to slow down ~2x to 1.16x/year to match your 1.4x number. I do think hardware gains will slow somewhat but 1.16x is below what I would bet. Taking my actual medians, I think i’m at like 1.3x (production) x 1.2x (hardware) = 1.56x/year so more like a 2.8x slowdown, not 3.7x slowdown.
So resolving the discrepancy, it seems like my model is basically saying that your model overestimates the slowdown because it assumes profit margins stay fixed, but instead under slowing investment growth these should collapse? That feels like it doesn’t fully explain it though since it seems like it should be a one time fall (albeit a big one). Maybe in the longer term (i.e. post-2035) I agree with you more and my 1.3x production number is too bullish.
A brief history of things that have defined my timelines to AGI since learning about AI safety <2 years ago
Bio anchors gave me a rough ceiling around 1e40 FLOP for how much compute will easily make AGI.
Fun with +12 OOMs of Compute brought that same ‘training-compute-FLOP needed for AGI’ down a bunch to around 1e35 FLOP.
Researching how much compute is scaling in the near future.
At this point I think it was pretty concentrated across ~1e27 − 1e33 flop so very long tail and something like a 2030-2040 50% CI.
The benchmarks+gaps argument to partial AI research automation.
The takeoff forecast for how partial AI research automation will translate to algorithmic progress.
The trend in METR’s time horizon data.
At this point my middle 50% CI is like 2027 − 2035, and would be tighter if not for a long tail that I keep around just because I think it’s have a bunch of uncertainty. Though I do wish I had more arguments in place to justify the tail or make it bigger, ones that compete in how compelling they feel to me to the ones above.
Thanks for linking. I skimmed the early part of this post because you labelled it explicitly as viewpoints. Then I see that you engaged with a bunch of arguments about short timelines, but they are all pretty weak/old ones that I never found very convincing (the only exception is that bio anchors gave me an early ceiling early on around 1e40 FLOP for compute needed to make AGI). Then you got to LLMs and acknowledged:
The existence of today’s LLMs is scary and should somewhat shorten people’s expectations about when AGI comes.
But then gave a bunch of points about the things LLMs are missing and suck at, which I already agree with.
Aside: Have I mischaracterized so far? Please let me know if so.
So, do you think you have arguments against the ‘benchmarks+gaps argument’ for timelines to AI research automation, or why AI research automation won’t translate to much algorithmic progress? Or any of the other things that I listed as ones that moved my timelines down:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch to around 1e35 FLOP
Researching how much compute is scaling.
The benchmarks+gaps argument to partial AI research automation
The takeoff forecast for how partial AI research automation will translate to algorithmic progress.
The recent trend in METR’s time horizon data.
But instead of discussing the crux of which system is relevant (which has to be about details of recursive self-improvement), only the proponents will tend to talk about software-only singularity, while the opponents will talk about different systems whose scaling they see as more relevant, such as the human economy or datacenter economy.
Totally agree! Thank you for phrasing it elegantly. This is basically what I commented on Ege’s post yesterday, I asked him to engage with the actual crux and make arguments about why the software-only singularity is unlikely.
There’s an entire class of problem within ML that I would see as framing problems and the one thing I think LLMs don’t help that much with is framing.
Could you say more about this? What do you mean by framing in this context?
There’s this quote I’ve been seeing from Situation Awareness that all you have to do is “believe in a straight line on a curve” and when I hear that and see the general trend extrapolations my spider senses start tingling.
Yeah that’s not really compelling to me either. SitA didn’t move my timelines. Curious if you have engaged with the benchmarks+gaps argument to AI R&D automation (timelines forecast), and then the AI algorithmic progress that would drive (takeoff forecast). These are the things that actually moved my view.
Thanks for the link, that’s compelling.
By far the most top-voted answer was one arguing that we might get AI to substantially accelerate AI progress because a particular AI research engineering benchmark looks like it will get saturated within a couple of years.
The list of things I see as concrete arguments that have moved down my timelines include exactly this!
But this is again assuming that good performance on a benchmark for AI research engineering actually translates into significant real-world capability.
...and I think this characterization is importantly false! This timelines forecast does not assume that. It breaks things down into gaps between benchmarks and real-world capability and tries to forecast how long it will take to cross each.
The benchmarks also do not take into account the fact that the vast majority of them measure a model’s performance in a situation where the model is only given one task at a time, and it can completely focus on solving that...
Agree that there are many such ‘gaps’! Would be curious to hear if you think there are important ones missing from the timelines forecast or if you have strong views that some of them will be importantly harder!
Furthermore, the most advanced reasoning models seem to be doing an increasing amount of reward hacking and resorting to more cheating in order to produce the answers that humans want. Not only will this mean that some of the benchmark scores may become unreliable, it means that it will be increasingly hard to get productive work out of them as their intelligence increases and they get better at fulfilling the letter of the task in ways that don’t meet the spirit of it.
Thanks for this! This is a good point. Do you think you can go further and say why you think it will be very hard to fix in the near term, so much so that models won’t be useful for AI research?
I agree re Leopold’s piece, it didn’t move my timelines.
A brief history of the things that have most collapsed my timelines down since becoming aware of AI safety <2 years ago:
Fun with +12 OOMs of Compute IMO, a pretty compelling writeup that brought my ‘timelines to AGI uncertainty-over-training-compute-FLOP’ down a bunch
Generally working on AI 2027, which has included
Writing and reading the capabilities progression where each step seems plausible.
Researching how much compute is scaling.
Thinking about how naive and limiting current algorithms and architectures seem, and what changes they are plausibly going to be able to implement soon.
The detailed benchmarks+gaps argument in the timelines forecast.
The recent trend in METR’s time horizon data.
Talked with some ZKP experts. The problem is that you need to generate the proof for each token in the context, which adds several OOMs. Apparently there are forthcoming solutions to this though.