Broad Timelines
No-one knows when AI will begin having transformative impacts upon the world. People aren’t sure and shouldn’t be sure: there just isn’t enough evidence to pin it down.
But we don’t need to wait for certainty. I want to explore what happens if we take our uncertainty seriously — if we act with epistemic humility. What does wise planning look like in a world of deeply uncertain AI timelines?
I’ll conclude that taking the uncertainty seriously has real implications for how one can contribute to making this AI transition go well. And it has even more implications for how we act together — for our portfolio of work aimed towards this end.
AI Timelines
By AI timelines, I refer to how long it will be before AI has truly transformative effects on the world. People often think about this using terms such as artificial general intelligence (AGI), human level AI, transformative AI, or superintelligence. Each term is used differently by different people, making it challenging to compare their stated timelines. Indeed even an individual’s own definition of their favoured term will be somewhat vague, such that even after their threshold has been crossed, they might have trouble specifying in which year it happened.
Many commentators have suggested this makes terms such as AGI useless, but I don’t think that is right.
I like to think of it in terms of a group of hikers seeing a mountain in the distance, towering up into the clouds and beyond, with its snowy peak catching the sun’s light. They talk animatedly about how amazing it would be to climb so high that they are inside a cloud. Or imagine being above the clouds, looking over them like an angel. After many hours of climbing, they notice there is a faint haze. Are they inside the cloud now? The mist gradually gets thicker until they can only see 10 metres ahead. Are they inside it now? Then it drops to 9 metres. Then 8. Then visibility starts to increase again. After an hour there is only the slightest haze. Are they above the clouds now? Another 30 minutes and there is no haze, and they can all agree they are above the clouds.
It is clear that at some point they were inside the cloud and sometime later were above it. And it is clear that these were sensible and useful concepts. For example, they took precautions like roping themselves together for the journey through the cloud due to the low visibility and took cameras with them because they knew they could take beautiful photos above the clouds. A lack of sharp boundaries doesn’t make these concepts useless. But they were admittedly a lot more useful when the hikers were on the ground, planning their route, and a lot less useful in the debatable boundary zones.
I think of AGI (and human-level intelligence) as the cloud, and superintelligence as being above the cloud. They are useful concepts, despite their vagueness. But they’re markedly less useful when you get close to them.
So I think that forecasting when we’ll reach some threshold for advanced, game-changing AI makes sense. Albeit there is some inherent uncertainty due to the vagueness of the ideas, and we have to be careful when comparing our estimates to make sure we’re talking about the same version of these concepts.
Regarding AGI, it’s already getting a bit misty. In February there was a piece in Nature arguing that the current level of frontier AI should count as AGI. I’d set the bar a bit higher than that, but I agree it is already debatable whether we’re in the cloud.
For my purposes, I think the key threshold is when the system is capable enough that there are dramatic changes to the world — civilisational changes. For example, the point where AI could take over from humanity were it misaligned, or it has made 50% of people permanently unemployable, or has doubled the global rate of technological progress. Something like that. The reason I pick this point is that I think it is the one that matters most for decision-relevant planning of our strategies and careers. For many purposes we’d want our plans to pay off before we reach that point, and plans that reach fruition afterwards are likely to be significantly disrupted. I’ll refer to this as transformative AI and will make sure to show what rubric other people are using when they give their own timeline numbers.
Short vs Long Timelines
Discussions about timelines are usually framed as a debate between short timelines vs long timelines.
One of the most prominent supporters of very short timelines is Dario Amodei, CEO of Anthropic. In January 2025 he said:
Making AI that is smarter than almost all humans at almost all things will require millions of chips, tens of billions of dollars (at least), and is most likely to happen in 2026-2027.
A month later, he clarified:
Possibly by 2026 or 2027 (and almost certainly no later than 2030), the capabilities of AI systems will be best thought of as akin to an entirely new state populated by highly intelligent people appearing on the global stage—a ‘country of geniuses in a datacenter’—with the profound economic, societal, and security implications that would bring.
At the other end, a good example of long timelines is Ege Erdil, Co-founder of Epoch AI, whose median time for the ‘full automation of remote work’ is 2045 — 20 years away.
While experts continue to disagree on when AI will start having transformative impacts, they are clearly not stubbornly ignoring the evidence. For as Helen Toner explained in her great essay: ‘Long’ timelines to advanced AI have gotten crazy short. Before ChatGPT, short timelines used to mean something like ‘10 to 20 years, so since it could take a long time to prepare, we should start now’. Long timelines used to mean ‘there was no sign AGI will happen in the next 30 years, if it happened this century at all, so it is premature to do any work related to controlling advanced AI’. But now we see short timelines like Dario Amodei’s with genius level AI ‘almost certain’ to happen within the next 5 years, and many staunch proponents of long timelines are now saying we’ll reach human-level in just 10 or 20 years.
Here’s a nice graph 80,000 Hours put together of how the average forecasted time until AGI on the Metaculus prediction site has shortened from about 50 years to about 5 years in just a 5-year window:
Broad Timelines
So everyone is updating on the evidence and shortening their timelines, yet substantial disagreement remains.
This is often framed as a debate: that we should be trying to assess who is right — whether timelines really are short or long (or medium). People pick winners, affiliate with one side or the other, and rub it in whenever the latest evidence favours their preferred camp.
My central claim today is that for most of us, that is the wrong frame. You should have neither short timelines nor long timelines — but broad timelines. That is:
The correct epistemic response to the lasting expert disagreement is to have a broad distribution over AI timelines.
First, there is too much disagreement among very smart and informed people for it to be reasonable to have a narrow range of possible years. You would need to ascribe very little chance to some of your epistemic peers seeing things more clearly than you do, when that actually happens half the time. Moreover, a lot of these people are coming from different fields, bearing diverse insights, evidence, and time-tested heuristics that no single individual is in a good position to judge.
And second, many of these people themselves have a broad distribution over AI timelines. For example, take Daniel Kokotajlo. He is one of the authors of AI 2027 and is known as a leading figure in the short timelines camp. A few years back, his median date for AI systems “able to replace 99% of current fully remote jobs” was 2027, hence the name of the scenario. Though his timelines have lengthened a little and by the time they were writing it, 2027 had become more of an illustrative early scenario rather than his point where it was 50% likely to have arrived.
Kokotajlo has done a great job of being extremely transparent about his timelines, showing his predictions (along with their uncertainty) for a variety of different levels of powerful AI. Here is his current probability distribution for when we will have an AI system that is “At least as good as top human experts at virtually all cognitive tasks”:
His distribution has its peak (the mode) in 2028, but because the distribution is heavily skewed towards the right, there is only a 27% chance of it happening by that point. His median year is 2030. And his 80% interval (from the 10th to 90th centile) is from 2027 to some point after 2050.
This is a broad distribution. I think someone’s 80% interval is a decent way of expressing the range of times they think are credible. Here Kokotajlo is saying that it will likely happen between 1 and 25 years from now, but that there is a 1 in 5 chance that it doesn’t even fall into that wide range.
He’s not the only one with such a broad distribution. Here are the forecasts of Daniel Kokotajlo, Ajeya Cotra, and Ege Erdil from 2023, forecasting: “In what year would AI systems be able to replace 99% of current fully remote jobs?”:
Note that all three have the same kind of shape, just stretched differently. And despite their very different medians they actually have a lot of overlap (which this transparent shading brings out). This shows both that each expert has a broad distribution and that the expert community on the whole has an even broader one. Indeed, I think you could do a lot worse than just taking a mixture model of these three experts’ views. Interestingly, since 2023, Kokotajlo’s distribution has shifted to the right and Erdil’s to the left.
Here’s an illustrative distribution for AGI timelines used by Ben Todd of 80,000 Hours:
Dwarkesh Patel reproduced it in his post about AI timelines, saying that it pretty much represented his own uncertainty, giving his median date of 2032 for AI that “learns on the job as easily, organically, seamlessly, and quickly as a human, for any white-collar work.”
Here is Metaculus’s current community estimate for when AGI will be developed. Synthesizing the community’s collective uncertainty, it is very broad and has this same characteristic shape:
Here is Epoch AI’s summary of leading estimates of AI timelines from 2023:
These look a bit different as they are represented as cumulative probabilities of reaching transformative AI by a given time. But they are all very broad. Take a look at the range of years between when they cross 10% to when they cross 90%. Every single one has an 80%-interval at least 50 years wide.
What about researchers working on AI capabilities? Grace et al surveyed thousands of AI researchers who were presenting at their top academic conferences. They surveyed the researchers in 2022 (blue) and 2023 (red) about when “unaided machines can accomplish every task better and more cheaply than human workers”:
You can see the wild variation in individual forecasts (the thin lines) and that the timelines became about 30% shorter in a single year. But vast uncertainty remains. The aggregate community forecasts (the thick lines) have 80% intervals ranging from years to centuries.
I think everyone should have a distribution that is roughly this shape. Here’s mine:
It is for transformative AI, loosely defined as AI that would be powerful enough to take over the world were it misaligned, and which is doubling the rate of scientific and technological progress. It’s a similar shape to Kokotajlo’s, but broader, with a median of 2038 and an 80% interval ranging from 3 years to 100 years.
Let’s return to where we started, with Daniel Kokotajlo’s distribution for AI that is “At least as good as top human experts at virtually all cognitive tasks”:
While we often express our timelines as single numbers (such as the mode or the median), I don’t think that’s a helpful approach here. Look at that graph. What number sums it up? Its only real feature is the peak, but Kokotajlo is saying it is unlikely to happen by then (just a 27% chance). The median is often a better number to give, but here it is at a relatively undistinguished point on the graph (in 4 years’ time) and saying ‘4 years’ would obscure his point that he thinks there is a 10% chance it is within 1 year and a 10% chance it is beyond 25 years.
I think that if he talked through what he actually means by this distribution with a smart policy maker, they would finally get it and say:
Oh, so you are saying you have no idea when it will happen — it could be next year, or it could be 6 presidential terms from now. And you’re saying there is a 1 in 5 chance it isn’t even in that range.
I think that’s actually a pretty good summary, and it would sum up my own distribution as well. While ‘no idea when it will happen’ is underselling the information contained in this distribution, it is a much better summary than ‘4 years’ which would be understood by almost everyone as something like ‘between 3 and 5 years’. While academics might hope people interpret a named year as the median time, most people interpret it as the moment they are allowed to start complaining the predicted event hasn’t happened yet.
Indeed, these distributions are so hard to sum up with a single number, that I think a substantial amount of disagreement on timelines stems from people describing different parts of the same elephant. For example, both AI boosters and those concerned with existential risk talk a lot about short timelines because ‘we could see the world transformed in just a few years’ time’. It isn’t that they think we will see that, but that it is big if true, and has a decent chance of being true. In contrast, more conservative voices tend to focus on later years saying ‘it is more likely that it will take 10 to 20 years, than that it will take just a few’ (focusing on straight probability without weighting by importance or leverage).
Both of these can be true at the same time. Both are true on my own distribution.
A particular danger in communicating timelines with a single number is that it raises the chance that this named year will come and go without incident, and the people who mentioned it (or the wider community they are part of) will be written off as having a false or discredited view. I think we’re going to see some of this come 2027 due to the vast number of people who heard about that scenario, combined with the fact that so many media outlets reported it as a sharp prediction, rather than as it was intended: an important illustrative scenario.
As well as being bad for communication, compressing one’s uncertainty into a single number would be very bad for your own planning.
For example Kokotajlo’s distribution implies a 28% chance transformative AI will happen during the current presidential term, a 35% chance it will happen in the next term, a 13% chance it will be the one after that, with 24% left over spread among ever more distant terms:
These are very different scenarios and it would clearly be a mistake to just act as if the second one were correct since it is the most likely. That would eliminate the possibility of hedging against transformative AI coming soon, and of taking advantage of worlds where it comes late.
Implications
Rather than attempting to adjudicate which length of timelines is correct, I think we should be taking the frame of how to act (or plan) under deeply uncertain timelines.
That is, we should be treating this as an exercise in rational decision-making under uncertainty — in a situation where the stakes are high and the uncertainty is vast.
Let’s unpack some of the implications of this frame.
We’ll start with two mistakes that are all too common in the policy world.
First, uncertainty about AI timelines isn’t an excuse to just believe whichever timeline you want, so long as it is within the credible range. Sadly, I think many government ministers are likely to take this approach if an expert explains this broad uncertainty to them. While they would be right that the evidence isn’t sufficient to disprove their preferred timeline, it would be irresponsible of them to not allow for other credible possibilities. That would be like a mayor hearing there is a 20% chance the volcano next to their town erupts next year and feeling that they can continue to act as if it won’t, since it not erupting is also found credible by the experts. Uncertainty isn’t an excuse to assume a plausible outcome of your choice will occur, it is more that rationality requires you to respect every plausible outcome.
Second, we can’t just wait until the uncertainty is resolved. Sometimes that works, but here we know the uncertainty is very unlikely to be resolved until the events are upon us. At that stage it will be too late to enact all but the most knee-jerk responses. So feeling that the cloud of uncertainty gives you permission to delay acting is tantamount to committing to choose one of the bluntest and least effective options available.
Instead, we are going to need to act under uncertainty, taking into account the full range of credible possibilities.
How can we do that?
Hedging
A natural and important idea is that of hedging against transformative AI coming soon —while we are least prepared. We could do that by shifting our portfolio of activities (or your individual contribution to humanity’s portfolio) to focus somewhat more on short timelines than the raw probabilities would warrant.
This makes a lot of sense. I strongly recommend governments, civil society, and academics do more to hedge against transformative AI coming early.
Though when it comes to the communities of professionals already working on helping the AI transition go well, I think they are already hedging strongly against early transformative AI. Indeed, there is even a risk that they are going beyond mere hedging, and are actively betting on it coming early. I’m not sure, as it is hard to know the full portfolio of work.
One certainly sees many more pleas for work aimed at very short timelines than for long timelines. But there are also strong reasons to consider long timelines in our planning, and ways in which work aimed at long timelines can also be extremely high leverage.
Let’s look at two key things that happen when timelines are longer.
A Different World
In longer timelines, AI arrives in a world that doesn’t look like today. The longer it is until transformative AI appears, the more different the world will be at that key moment.
As a baseline, suppose it arrives soon, in 2028. Things will definitely be different to today, but we’d expect many of the broad brushstrokes to be similar. We would likely have the same US president, the same major players, the same main technologies. If transformative AI arrived within just two years, I’d bet it was something like the AI 2027 story where a lab recklessly got recursive self-improvement going.
Now suppose transformative AI arrives in 2035. That is not this presidential term or even the next one, but the one after that. Who knows who’d be in power, or what state the US would be in. The nine years would likely have seen major changes in the core technologies of AI (9 years before now there were no LLMs or transformers). We could well have different leading AI companies, perhaps as a result of a bubble having burst and taken out the overextended first-movers.
By 2035, export controls may well have backfired, helping China get ahead on chips by incentivising them to build out their own chip industry and giving them 13 years to get good at it. This was a key dynamic the White House considered while drafting the export controls, but they were focused on shorter timelines… By 2035, China may have also invaded Taiwan, depriving the West of their biggest source of chips.
By 2035, there may be double-digit unemployment from increasingly powerful AI systems and public sentiment about AI could be very strong. The Overton window for AI regulation will be in a very different place.
As may be the geopolitical order. The last nine years has seen the invasion of Ukraine, the increasing isolation of the US and a global pandemic. Another nine years could see a similar amount of change.
And if we haven’t played our cards right, those of us working on avoiding catastrophic risks from AI may have also lost a lot of power, with our ideas about AI risk being seen as discredited since so many years have passed without the truly transformative effects we were talking about.
In short, the longer the timelines the more different things will be — both in some systematic, predictable ways, and just from random diffusion and chaos. So taking longer timelines seriously means:
Being more open to approaches that wouldn’t work in the world as it is today,
Being less excited about approaches that are tailored to the specifics of the today’s world,
Being less happy to compromise your values to appeal to those currently in control of companies and governments,
Being less willing to say things that will make people feel our position is discredited if we end up in a long timeline world,
And spending less time following the daily news about what has just happened in AI or who is ahead.
Longterm Actions
There are many kinds of things people can work on that can pay off handsomely, but only after a number of years. Things like:
Founding and nurturing a new research field
Founding an organisation or company
Building a movement or community
Writing a book
Foundational research
Completing a PhD
A major career change
Climbing the ladder in a large organisation or government
Training promising students in AI Safety or AI Governance
If you just consider your impact during the next three years, most of these will be beaten by other shorter-term options. But as the years climb, longer-term options can have very high value. They aren’t always best, but for the right people or the right opportunities, they can be extremely impactful.
When I was a grad student, I realised how much good I could achieve if I donated much of my income over my career to help those in the poorest countries. And the more I thought about it, the more I thought I should start something — an organisation — to help other people to do this too. So Will MacAskill and I launched Giving What We Can in 2009. 17 years later, more than 10,000 people have joined us, having thousands of times as much impact as if I’d carried on alone.
This kind of compounding growth is one of the major ways that longer term projects can have very large multipliers, giving us a very big boost to our impact if timelines are in fact long.
Starting new fields can be similar. When I first met Allan Dafoe 10 years ago, I didn’t know what he was talking about when he spoke of ‘AI governance’ — a new field he was trying to found. Now it is a burgeoning field, with hundreds of practitioners, who are in high demand from many different governments.
When I started writing The Precipice, I wasn’t sure I should, because I thought AGI might just be too close. But as it turns out, there was time to write it and for it to have a real impact. I’m really glad I did, as I meet so many amazing people working on the biggest risks who tell me it was reading The Precipice that inspired them to do so. I think it is one of the best things I’ve done.
After it came out, I used to think that there just wasn’t enough time to write a further book — that we were really too close to the critical moment. We might be, but I think I was mistaken about the strength of this argument. The time horizon for a book to have real impact is about 5 years (time to plan the book, win a book deal, write the book, wait for publishers to publish it, then wait a year or more before it has sufficient impact in the world).
But I only think there is about a 1 in 5 chance of transformative AI coming in the next 5 years. So while a book may come out too late, that is only a 1 in 5 chance, leaving a book project with 80% as much expected value as I’d have naively calculated. So while there is a 1 in 5 chance I’d be kicking myself, on my views about AI timelines there isn’t actually that much of a haircut in expected value due to the chance it is too late.
That said, the chance of transformative AI arriving before your work pays off is only one factor affecting whether you should do work aiming at short or long timelines. Another is that AI safety and governance are likely to be more neglected now than they will be later. This creates an extra multiplier for the value of direct work in these areas now, and in some cases is a larger effect than the chance your work comes to fruition after transformative AI.
Overall, I think that longer term projects do get down-weighted by these considerations, but their advantages sometimes outweigh that — especially if they are shooting for a very big payoff. I’d guess that if someone looked at their options and thought the best option was one that took 5 to 10 years to pay off, then about half the time it would remain their best option even after taking AI timelines into consideration. After all, it is not uncommon for your best option to be several times better than your second best.
So I think the community of people working on transformative AI are likely underrating types of work that need five or more years in order to pay off. The ideal portfolio of activities aimed at making the AI transition go well should include a number of things that really help us succeed in worlds where we get longer to try.
But I want to stress that none of this implies we can slack off.
We’re in a race against AI timelines. It is just that we don’t know if that race is a sprint or a marathon. In either case, time is of the essence.
Conclusions
We have seen that there is substantial disagreement and uncertainty about when AI will start having transformative impacts on the world. This is because there just isn’t enough evidence to pin it down. My claim is that for the purposes of planning we should adopt neither short nor long timelines, but broad timelines:
The correct epistemic response to the lasting expert disagreement is to have a broad distribution over AI timelines.
Given this deep uncertainty we need to act with epistemic humility. We have to take seriously the possibility it will come soon and hedge against that. But we also have to take seriously the possibility that it comes late and take advantage of the opportunities that would afford us. The world at large is doing too little of the former, but those of us who care most about making the AI transition go well might be doing too little of the latter.
We need to take more seriously the possibility that the world will look very different at that time, which should broaden our own Overton windows about what kinds of plans could succeed. And we shouldn’t be ruling out all actions which take a long time to pay off. Even if they wouldn’t help in short timelines worlds, some actions more than make up for this with substantial impacts if timelines are long.
Funders, career advisors, and movement builders should be thinking about this with regards to how we act as a community: to the shape of the whole portfolio of work aimed at effectively improving the world. And each of us should be reflecting on what this deeply uncertain timing means for planning our own contributions over the years to come.
I agree with many particular points in this post and the apparent thesis [1] , but also think most people [2] should focus on short timelines (contrary to the apparent implication of the post). The reasons why are:
Short timelines have more leverage. This isn’t just because of more neglectedness now, but also because: (1) it’s easier to target approaches towards shorter timelines where less has changed, (2) short timelines are riskier (and I think riskier worlds are more leveraged for most interventions, this is sensitive to my views on risk and the most leveraged interventions), and (3) it’s easier to operate in near mode when targeting short timelines and I expect this has a bunch of benefits (mostly from psychological / cognitive bias perspective).
I put sufficiently high probability on short timelines: maybe 25% in <2.5 years to full AI R&D automation and 50% in <5. I don’t think deference to other experts shifts me towards longer timelines by much. [3] I think there are good arguments for this view, though I certainly agree there isn’t consensus and the arguments aren’t that clear cut or legible.
I expect work explicitly focused on short timelines (across most areas) to transfer pretty well and generally not cause that much downside in longer timelines. I think the transfer in the other direction tends to look less good in practice. (To be clear, I think work focused on short timelines shouldn’t neglect thinking about downsides in longer timelines, I just think this is usually not that big of a deal.)
The counterargument I’m most sympathetic to is that (1) a high fraction of the work should be focused on “better futures” and (2) for better futures work, the leverage is higher in longer timelines. (I don’t currently agree with either of (1) or (2), but I’m very uncertain.)
Assuming the thesis is “our probability distribution should span a wide range (including Daniel’s distribution as an example of a wide range) and we should take this into account in our decision making.
Or at least most of the quality weighted labor supply.
I might have a small difference between these stated probabilities and my full all considered view including defering to others. To avoid deference cascades, I usually state probabilities somewhat closer to my non-deference view. (It’s hard to fully disentangle deference because my views are based on talking to a wide range of different people.) Post deference my distribution is a bit wider with a correspondingly longer median. But I don’t think this makes much difference either way and deference also pulls up my probability on very short timelines.
Hm. Skeptical of this. From my relative lay perspective, it sure seems like Anthropic and others use justifications like “This could be coming soon. On that assumption, we can get to the forefront and do our best to work out safety and do the right thing.” and then they push the forefront foreward. Which is bad to do.
Holy moly! What were your AGI timelines when you started writing?
This point seems obvious and commonsense, but I have frequently heard good rationalists talk as though the median is the estimate, rather than reasoning with the full uncertainty.
Taking the uncertainty seriously is difficult and stressful. Trying to do that is fighting a small Ugh field or motivated reasoning that tends to generate cope or hopium wishful logic.
Thanks for doing the full careful writeup.
This previous LessWrong article seems extremely relevant and basically sketches out an example of the rough “strategic portfolio” for AI risk that you are arguing for.
In line with some of my recent posts, I’m starting to think there is a lot of value in:
Clearly defining consensus group strategy (among LW/EA/CG, for example) on “making the future go well.” This should include rough estimates from a variety of respected sources, a diverse portfolio of interventions, and explicitly communicated uncertainty / epistemic humility.
Designing info-UI tools to facilitate that process. Enabling effective deliberation, strategy adjustment, and maybe most importantly: easy-to-use interfaces for the general public. The goal being to convey community beliefs and disagreements in a very transparent and easy to understand way.
This is intentionally unspecific but I have outlined a couple particular ideas in previous posts and will continue to crystallize my suggestions / explain why I think this area has potential.
The AI futures model and related ecosystem is a great start but is limited to a handful of thinkers (Daniel and Eli) and a specific subset of information (forecasting timelines). Their work has already been quite impactful (read by JD Vance apparently) – why not work hard to apply and scale good information-interface-design to broader community strategy?
Re criteria for AI being transformative in some sense, it might be useful to look at how historians determine sharp dates for fuzzy-edged eras, which at least sometimes seems to involve identifying key events across multiple domains. E.g. common criteria for the end of the Middle Ages include:
1453: Fall of Constantinople
1455: Gutenberg Bible (first major printed book)
1492: Columbus’s first voyage
1517: Luther’s 95 theses
Viz. in this case, key political, technological and cultural criteria. (For which the years could be averaged, say, to c.1480.)
So I actually like the cloud metaphor. But as you say
it’s debatable whether we’re in the cloud, which means that perhaps we should be forecasting other things than “AGI”.
And you do give some more specific milestones, like
but (1) “the point where AI could take over from humanity were it misaligned”, (2) “[the point where] it has made 50% of people permanently unemployable”, and (3) “[the point where it] has doubled the global rate of technological progress” seem to me plausibly quite distinct. I
expect[ETA[1]: find it plausible for] (1) to come years after (2) and (3); and (3) probably before (2), but I have substantial uncertainty here. Maybe you could get (2) before (3) if you really crank boring LLM training the right way + robotics. And maybe powerful optimization is out there to screw us, and (1) comes before (2) and (3). If I were to forecast it in a principled probabilistic way, I would end up with three distinct, but somewhat dependent probability distributions.Worth marking that it’s a log-x plot.
see the thread below for details
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster, [1] a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI [2]
Conclusion. if you somehow don’t have wildly superhuman AI at the beginning of this thought experiment, you will have it in at most
(On the surface, it looks like I’m saying “if P then Q”, where P is (2) AND (3) AND not-(1) (using your numbering of events), and Q is (1)-soon, so making a claim about the conditional in which P is true. But that’s not really what I mean to say. Like, to me P is true in some sorts of weird magic worlds carrying (let’s say)
Another pathway that I have substantial mass on: [the first learning setup in which novel mental structure creation / novel domain learning starts to work basically at all] produces a wildly superhuman AI. Like, stuff gets crazy inside a single training/growth process.
absent strong AI regulation
actually, i’d guess this is true even with “at most 1 year” instead of “at most 10 years”
In such a case I expect these AI researchers to pick all the low- and medium-hanging fruit at the then-current compute level/hardware technology, and then the algorithmic progress gets saturated until new-gen chips are produced in quality. Check this: https://www.lesswrong.com/posts/sGNFtWbXiLJg2hLzK
Since my Claim 1 is about the conceptual work input being 100x sped up, not some final output being 100x sped up, I’ll take you to be disagreeing with Claim 2. So the question is: is 10 years of thinking about AI algorithms followed by 1 month of retraining sufficient [to get from AI that causes
I think we should imagine the fruits at the beginning of this to not have been well-picked (supposing a crazy smart AI does not already exist).
Trusting Byrnes’s decomposition of the 7 year 600x nanochat cost improvement, that’s 6x from hardware and 100x from non-hardware. That would give some sort of baseline guess of
Byrnes explicitly does not include algo ideas that “are not about doing the same thing more efficiently, but rather about trying to do something different instead”. See Section 1.5 of his post. But these clearly should be included in our context here, and are majorly important imo. E.g., curating curricula, creating problems for oneself in a different way, coming up with good ways to reward problem-creation, creating more nested levels of problem-solving with their own rewards, coming up with other ways to make rewards denser / track progress better, creating tools for oneself, various IDA ideas (beyond those already mentioned), etc.. There are also various ways humans get smarter over centuries and over a lifetime that should also count for our purposes as “algo progress” if the AIs can carry them out, e.g. inventing+acquiring new concepts, questions, methods, and skills, and just knowing more.
In our scenario, coming up with an arbitrarily different new AI design is also legitimate, as long as this AI can be created/trained/grown in at most 1 month.
Tbh a lot of my belief that you get a lot of progress just comes from it being an extremely high-dimensional design space and there surely being lots of things one can do so much better in there.
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
Unless humans are strictly needed to orchestrate the AIs, but a world where they have thinking coherent enough to make massive algorithmic progress, but incoherent enough to pursue this competently, seems super weird, but, hey, maybe Moravec will come to bite us again!
You mean something like serial, uninterrupted, focused thinking, like a WBE of a very-high-g AI researcher that doesn’t need sleep?
this makes me want to ask: are you tracking the difference between the event “50
(to make sure we’re on the same page: in my view, this is unlikely to be a somewhat stable situation)
Yes, I am talking about 50% people being permanently unemployable, i.e., not being capable of doing any labor that someone would pay meaningful amounts for.
It seems to me that the crux between us is something like: I find a very jagged capability, “Moravec-ian” world plausible, i.e., AI can do lots/most of economically valuable stuff competently, with the amount of human oversight small enough to make 50% of humanity permanently unemployable, while still not being “human-level” on all axes and this remaining stable for a few years at least (which also touches on your claim 2, i.e., an AI that could do all this doesn’t yet exist).
But maybe I’m wrong, and you actually need to be way closer to “human-complete” to do all the boring economic tasks, and AI that is not near-human-complete would not be massively deployable to do them with minimal oversight.
I am now more uncertain, so I will somewhat revise my top-level comment.
i just meant the usual human AI algo research community doing this for 10 years
I realized I’m not sure how you define “50% of people permanently unemployable”. Surely it isn’t about global population? Is it about global labor force (which is ~45% of global population) or about developed countries only?
As of 2019, about a quarter of global labor force worked in primary agricultural production (mostly smallholder farmers who might only be impacted by AI indirectly, such as natural gas going to data centers instead of fertilizer plants) and half as much were employed in “off-farm segments of agrifood systems”. Surely people need to eat and those jobs are here to stay.
Please define specifically, 50% of which people in particular, and what does “permanently unemployable” mean exactly (for example, what about a laid-off white-collar worker who can return to the parents’ village and get a job at a local shop or school?)
We have ways to measure unemployment. It classifies some people as unemployed and classifies some people as labor force, and the fraction of the former divided by the latter is the unemployment rate, which today hovers around 5% (depending on the country). 50% of people are permanently unemployable if they are too useless to leave that category by the time they stop counting as participating in the labor force.
(Also, to be clear, Toby was the first one who used the phrase “50% of people permanently unemployable” without defining clearly what he meant by this, and I responded to it without reflecting that maybe it’s good to clarify (and then Kaarel did the same after me). So, I appreciate you pushing me for clarification, but depending on your interest, you might care more about what Toby means by that, not what I mean by that.)
The most common reason people stop counting as participating in the labor force is that they grow old and living off savings, passive income, pension and/or social benefits is better than continuing working, which we call a retirement. With global graying of the population, 50% of formerly working people will necessarily become permanently unemployable in this sense eventually even without the AI progress.
Also, note that Finland has ~10% unemployment rate and they are quite OK because of the social safety net. If AI was to be heavily taxed and these funds were used to support the population suffering job losses (implausible for the US indeed but plausible for Europe), even in absence of “strong AGI” people might choose not to work in order to get welfare without actually being unemployable.
(Yeah I do care but Toby has not left a single comment here)
This is very heartening for those of us who are too young or too low on resources to make contributions on short timelines in any situation except one of the most acute desperation; if a college student were to do all they could to stop ASI they could throw their life/career away to try and make a difference, but they only get one shot at that, so it better count or happen when things actually matter that much.
Otherwise, it is good to note that we can still contribute to hedging against long term scenarios by investing in those outcomes. I am going to think deeply on what long/medium term strategies could be the most impactful from my position.
I suspect that broad timelines are worse even than Cotra’s mistake of creating a broadly right model and including absurd parameter values into the model.
Are you saying we should have narrow timelines because it’s obvious when AI is going to arrive? Or that we should work harder to narrow our subjective probability distributions? Or something else?
I firmly believe that the OP’s author should have reduced the uncertainty at least to a Lifland-like estimate. Additionally, I struggle to understand most constraints related to broad timelines. Whatever the timelines are, our endgoal is to ensure that the ASI is either never created or aligned and aligned not to a dystopia. Preventing a misaligned ASI requires a leverage at least over actors as reckless as xAI, and preventing Intelligence Curse-like outcomes or AI-enabled dictatorships requires some influence over power struggles. Such an influence requires us to ensure that politicians occupying positions of power act to prevent risks, not to do things like destroying Anthropic for a refusal to participate in mass survelliance. But I don’t see any pathways except for infecting politicians with the right memes (think of IABIED’s attempt to flood politicians with calls, letters and e-mails or of the IABIED march) and placing infected people into higher-level positions.
Moreover, Kokotajlo’s timeline implies a 50% chance of TED-AI before Jan 2031 or before Oct 2032, Eli’s timeline implies a 50% chance of TED-AI before Feb 2035 or Apr 2036. Taken at face value, these estimates mean that p(TED-AI is created within the next 10 years) is around 50% (or, in Kokotajlo’s case, 62% or outright 73%), making a project requiring 20 years to be completed unlikely to have an effect.
IDK what you mean by “TED-AI” but, in case you haven’t noticed, Ord’s median seems to be 2038, which is like 2 or 3 years later than Lifland.
TED-AI is defined by Kokotajlo-Lifland as Top Expert Dominating AI. However, I struggle to understand the origins of @Toby_Ord’s distribution. I suspect that his sources for longer timelines are as hard to rely on as is Cotra’s heavily criticized estimate or the fact that “all the revenue growth in the industry has corresponded to a scaling up of the supply of inference compute so that revenue per H100 equivalent has remained fairly constant.” Unlike things like the Epoch Capabilities Index as dependent on training compute or the ARC-AGI leaderboard per money spent (which might imply that no possible CoT-based system is far more effective at ARC-AGI than Gemini 3 Flash and Gemini-3.1 Pro), Ergil’s argument doesn’t actually claim anything about capabilities of AI systems which don’t even exist yet.