Isn’t it kinda unreasonable to put 10% on superhuman coder in a year if current AIs have a 15 nanosecond time horizon? TBC, it seems fine IMO if the model just isn’t very good at predicting the 10th/90th percentile, especially with extreme hyperparameters.
I also don’t know how they ran this, I tried looking for model code and I couldn’t find it. (Edit: found the code.)
I don’t think the defense of being bad at 10% and 90% work because those are defined by the non-super-exponential scenarios, and they are detached from the crux of the problem. Those are the more honest numbers as they actually respond to parameter changes in ways at least attempting to be related to extrapolation.
40-45% of the population, who almost necessarily definethe median alone, have a honest, to-infinity singularity at 25-90 months that then gets accelerated a bit further by the secondary super-exponential factor, research progress multipliers. This basically guarantees to a median result in 2-5 years (honestly tighter than that unless you are doing nanoseconds), give or take the additive parameters, that has nothing to do with what people think the model is saying.
If the model is about “how far we are to super-human effort and are extrapolating how long it takes to get there”, a median-defining singularity that is not mentioned except in a table is just not good science.
For another look, here is a 15 millisecond-to-4 hour time horizon comparison just in the super-exponentiated plurality.
This is a prediction with output deviations representing maybe hundredth-SD movements over parameter changes of 10^6 (on the baseline parameter) in an estimation task that includes terms with doubly-exponentiated errors.
I don’t think something like this is defensible based on the justification given in the appendix, which is that forecasters (are we forecasting or or trend-extrapolating pre-super-coders?) think the trend might go straight to infinity before 10 doublings (only 1024x) because humans are better going from 1-2 year tasks than 1-2 month tasks.
Edit: as I am newbie-rate-limited may not be able to talk immediately!
Here is the pure singularity isolated (only the first super-exponential function applied) and the same with normal parameters, which mostly just hide what’s happening:
Thanks a lot for digging into this, appreciate it!
being bad at 10% and 90% work
Is the 10% supposed to say 50%?
If the model is about “how far we are to super-human effort and are extrapolating how long it takes to get there”, a median-defining singularity that is not mentioned except in a table is just not good science.
Yup, very open to the criticism that the possibility and importance of superexponentiality should be highlighted more prominently. Perhaps it should be highlighted in the summary? To be clear, I was not intentionally trying to hide this. For example, it is mentioned but not strongly highlighted in the corresponding expandable in the scenario (“Why we forecast a superhuman coder in early 2027”).
For another look, here is a 15 millisecond-to-4 hour time horizon comparison just in the super-exponentiated plurality.
copied from response to Ryan: Yeah I guess I think this is probably wrong. Perhaps one way to frame this would be that we would think there was a much lower chance of aggressive superexponentiality if we started at a lower time horizon, so we are sort of hardcoding the superexponentiality parameters to the current time horizon. This might not be the only issue though.
The superexponential in the simulation is implemented pretty crudely, if you have ideas on how to improve it I’d be interested, I didn’t spend a large amount of time on this. I think this is happening because it’s crudely implemented as each doubling taking 10% shorter, but this already leads to extremely low doubling times by the time the marginal doublings from the starting time horizon are added.
An obvious marginal thing I could do is to add uncertainty as to the shortening factor, though when we tested this and I eyeballed it it didn’t make that big of a difference (but I might have been mistaken). I also think this would ideally be implemented as some chance of going superexponential after each doubling rather than a chance of it happening only from the start, but didn’t have time to implement that.
I will defend the basic dynamics though of substantial probability of a superexponentiality leading to a singularity on an extended version of the METR time horizon dataset being reasonable. And I don’t think it’s obvious whether we model that dynamic as too fast or too slow in general.
Here is the pure singularity isolated (only the first super-exponential function applied) and the same with normal parameters, which mostly just hide what’s happening:
Yup, similar reaction to the above graph, I agree the shape looks wrong, interested in suggestions if you have them.
edit: actually I’m pretty confused about this last graph, what code exactly are you running? I don’t think I understand.
Thanks a lot for digging into this, appreciate it!
Of course, appreciate the response!
being bad at 10% and 90% work
Is the 10% supposed to say 50%?
This was a rebuttal to Ryan’s defense of the median estimates by saying that, well, the 10/90th might not be as accurate esp with weird parameters, but the 50th might be fine. So I was trying to show that the 50th was the heart of the problem in response.
Yup, very open to the criticism that the possibility and importance of superexponentiality should be highlighted more prominently. Perhaps it should be highlighted in the summary? To be clear, I was not intentionally trying to hide this. For example, it is mentioned but not strongly highlighted in the corresponding expandable in the scenario (“Why we forecast a superhuman coder in early 2027”).
Not implying intentional hiding! I honestly think that’s pretty irrelevant since the data are purely produced through non-reproducible intuited or verbally justified parameter estimates, so intentionality is completely opaque to readers, which is really the bigger core problem with the exercise. Imagine if climate researchers had made a model where carbon impact on the environment gets twice-to-infinity times larger in 2028 based on “that’s what we think might be happening in Jupiter.” Nobody here would care about intentionality and I don’t think we should care or think it matters most in this case either.
I will defend the basic dynamics though of substantial probability of a superexponentiality leading to a singularity on an extended version of the METR time horizon dataset being reasonable. And I don’t think it’s obvious whether we model that dynamic as too fast or too slow in general.
I think this is not true, but I don’t think it is worth litigating separately from the research progress super-exponential factor which is also modeled as already happening. so it is easeir (though I don’t think easy) to defend both individually based on small potatoes evidence of a speedup, but I can’t imagine how both could be defended at once, so that has to come first so we don’t double count evidence.
Here is the parameter setting for the “singularity-only” version:
eli_wtf:
name: "Eli_wtf 15 Min Horizon Now"
color: "#800000"
distributions:
h_SC_ci: [1, 14400] # Months needed for SC
T_t_ci: [4, 4.01] # Horizon doubling time in months
cost_speed_ci: [0.5, 0.501] # Cost and speed adjustment in months
announcement_delay_ci: [3, 3.01] # Announcement delay in months (1 week to 6 months)
present_prog_multiplier_ci: [0.03, 0.3] # Progress multiplier at present - 1
SC_prog_multiplier_ci: [1.5, 40.0] # Progress multiplier at SC - 1
p_superexponential: 0.45 # Probability of superexponential growth
p_subexponential: 0.1 # Probability of subexponential growth
se_speedup_ci: [0.05, 0.5] # UNUSED; 80% CI for superexponential speedup (added to 1)
sub_slowdown_ci: [0.01, 0.2] # UNUSED; 80% CI for subexponential slowdown (subtracted from 1)
se_doubling_decay_fraction: 0.1 # If superexponential, fraction by which each doubling gets easier
sub_doubling_growth_fraction: 0.1 # If subexponential, fraction by which each doubling gets harder
Then, split out the calculate_base_time function from the calculate_sc_arrival_year function and plot those results (here is my snippet to adjust the base time to decimal year and limit to the super-exponential group: base_time_in_months[se_mask] / 12 + 2025 ). You’ll have to do some simple wrangling to expose the se_mask and get the plotting function to work but nothing substantive.
I honestly think that’s pretty irrelevant since the data are purely produced through non-reproducible intuited or verbally justified parameter estimates, so intentionality is completely opaque to readers, which is really the bigger core problem with the exercise.
Okay, we have a more fundamental disagreement than I previously realized. Can you tell me which step of the following argument you disagree with:
It’s important to forecast AGI timelines.
In order to forecast AGI timelines while taking into account all significantly relevant factors, one needs to use intuitive estimations for several important parameters.
Despite (2), it’s important to try anyways and make the results public to (a) advance the state of public understanding of AGI timelines and (b) make our reasoning for our timelines more transparent than if we gave a vague justification with no model.
Therefore, it’s a good and important thing to make and publish models like ours
Imagine if climate researchers had made a model where carbon impact on the environment gets twice-to-infinity times larger in 2028 based on “that’s what we think might be happening in Jupiter.”
A few thoughts:
Climate change forecasting is fundamentally more amenable to grounded quantitative modeling than AGI forecasting, and even there my impression is that there’s substantial disagreement based on qualitative arguments regarding various parameter settings (though I’m very far from an expert on this).
Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).
I’d be curious if you have the same criticism of previous timelines forecasts like Bio Anchors.
I don’t understand your analogy re: Jupiter. In the timelines model, we are trying to predict what will happen in the real world on Earth.
I think this is not true, but I don’t think it is worth litigating separately from the research progress super-exponential factor which is also modeled as already happening. so it is easeir (though I don’t think easy) to defend both individually based on small potatoes evidence of a speedup, but I can’t imagine how both could be defended at once, so that has to come first so we don’t double count evidence.
I don’t understand why you think we shouldn’t account for research progress speedups due to AI systems on the way to superhuman coder. And this is clearly a separate dynamic from the trend being possibly superexopnential without these speedups taken into account (which, again, we already see some empirical evidence for this possibility). I’d appreciate if you made object-level arguments against one or both of these factors being included.
Then, split out the calculate_base_time function from the calculate_sc_arrival_year function and plot those results (here is my snippet to adjust the base time to decimal year and limit to the super-exponential group: base_time_in_months[se_mask] / 12 + 2025 ). You’ll have to do some simple wrangling to expose the se_mask and get the plotting function to work but nothing substantive.
I honestly think that’s pretty irrelevant since the data are purely produced through non-reproducible intuited or verbally justified parameter estimates, so intentionality is completely opaque to readers, which is really the bigger core problem with the exercise.
Okay, we have a more fundamental disagreement than I previously realized. Can you tell me which step of the following argument you disagree with:
It’s important to forecast AGI timelines.
In order to forecast AGI timelines while taking into account all significantly relevant factors, one needs to use intuitive estimations for several important parameters.
Despite (2), it’s important to try anyways and make the results public to (a) advance the state of public understanding of AGI timelines and (b) make our reasoning for our timelines more transparent than if we gave a vague justification with no model.
Therefore, it’s a good and important thing to make and publish models like ours
I have nothing against publishing models like this! I think the focus should be on modeling assumptions like super-exponentiality and treated as modeling exercises first-and-foremost. Assumptions that highly constrain the model should be first and foremost rather than absent from publicly facing write-ups and only in appendices. Perturbation tests should be conducted and publicly shown. It is true that I don’t think this represents high-quality quantitative work, but that doesn’t mean I don’t think any of it is relevant, even if I think the presentation does not demonstrate a good sense of responsibility. The hype and advertising-ness of the presentation is consistent, and while I have been trying to avoid particular comment thereon, it is a constant weight hanging over the discussion. Publication of a model and breathless speculation are not the same, and I really do not want to litigate the speculative aspects but they are the main thing everyone is talking about including in the New York Times.
I think this is not true, but I don’t think it is worth litigating separately from the research progress super-exponential factor which is also modeled as already happening. so it is easeir (though I don’t think easy) to defend both individually based on small potatoes evidence of a speedup, but I can’t imagine how both could be defended at once, so that has to come first so we don’t double count evidence.
I don’t understand why you think we shouldn’t account for research progress speedups due to AI systems on the way to superhuman coder. And this is clearly a separate dynamic from the trend being possibly superexopnential without these speedups taken into account (which, again, we already see some empirical evidence for this possibility). I’d appreciate if you made object-level arguments against one or both of these factors being included.
I think this has a relatively simple crux
Research progress has empirically slowed according to your own source of EpochAI (Edit: as a percentage of all progress which is the term used in the model. It is constant in absolute terms)
You assume the opposite which would explain away the METR speed-up on its own
So justifying the additional super-exponential term with the METR speed-up is double counting evidence
Then, split out the calculate_base_time function from the calculate_sc_arrival_year function and plot those results (here is my snippet to adjust the base time to decimal year and limit to the super-exponential group: base_time_in_months[se_mask] / 12 + 2025 ). You’ll have to do some simple wrangling to expose the se_mask and get the plotting function to work but nothing substantive.
Could you share the code on a fork or PR?
I can drop my versions of the files in a drive or something, I’m just editing on the fly, so they aren’t intended to be snapshotted, but here you go!
Imagine if climate researchers had made a model where carbon impact on the environment gets twice-to-infinity times larger in 2028 based on “that’s what we think might be happening in Jupiter.”
A few thoughts:
Climate change forecasting is fundamentally more amenable to grounded quantitative modeling than AGI forecasting, and even there my impression is that there’s substantial disagreement based on qualitative arguments regarding various parameter settings (though I’m very far from an expert on this).
Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).
I’d be curious if you have the same criticism of previous timelines forecasts like Bio Anchors.
I don’t understand your analogy re: Jupiter. In the timelines model, we are trying to predict what will happen in the real world on Earth.
4. The analogy is that the justification uses analogy to humans which are notably not LLMs. If it is purely based on the METR “speed-up” that should be made more clear as it is a (self-admittedly) weak argument.
(I offered to call with Peter because I’m having trouble understanding his positions over text-based discussion. edit: looks like we will call)
I will briefly say re:
Assumptions that highly constrain the model should be first and foremost rather than absent from publicly facing write-ups and only in appendices. Perturbation tests should be publicly shown as should variances in assumptions. It is true that I don’t think this represents high-quality quantitative work, but that doesn’t mean I don’t think any of it is relevant, even if I think the presentation does not demonstrate a good sense of responsibility. The hype and advertising-ness of the presentation is consistent, and while I have been trying to avoid particular comment thereon, it is a constant weight hanging over the discussion. Publication of a model and breathless speculation are not the same, and I really do not want to litigate the speculative aspects but they are the main thing everyone is talking about including in the New York Times.
I agree in an ideal world the timelines model would be more rigorous with better structure, more ablation studies, more justifications, etc. However this would have taken a lot longer and my guess is that we would have ended up with similar forecasts in terms of their qualitative conclusions about how plausible AGI in/by 2027 is (though of course, I’m not certain). We didn’t want to let perfect be the enemy of the good/useful (i.e. good/useful in our opinion; seems like the crux may be you disagree that it’s useful).
Apologies if any part of the media stuff seemed like it overclaimed regarding our work.
“we would have ended up with similar forecasts in terms of their qualitative conclusions about how plausible AGI in/by 2027 is” if this is with regard to perturbation studies of the two super-exponential terms, I believe based on the work I’ve shared in part here it is false, but happy to agree to disagree on the rest :)
I’m curious to hear what conclusions you think we would have came to & should come to. I’m skeptical that they would have been qualitatively different. Perhaps you are going to argue that we shouldn’t put much credence in the superexponential model? What should we put it in instead? Got a better superexponential model for us? Or are you going to say we should stick to exponential?
Thanks for engaging, I’m afraid I can’t join the call due to a schedule conflict but I look forward to hearing about it from Eli!
Ah, I was trying to avoid implying a lack of integrity in the forecasting effort, and as a result ended up implying knowing other things about your mental state.
To restate: I do not think a forecaster not previously committed to achieving a result with a median around that point would have, after doing a perturbation analysis that would display how dominant the two superexponential terms are in predetermining that specific outcome, presented those conclusions without hammering the point that those two superexponential terms are completely determining the topline result and that no other parameters make a meaningful contribution, whether forecasts of compute availability, current capabilities, capabilities required for SC, or otherwise.
Sorry if my previous comment implied something else!
Assumptions that highly constrain the model should be first and foremost rather than absent from publicly facing write-ups and only in appendices.
Strongly agree — cf. nostalgebraist’s posts making this point on the bio anchors and AI 2027 models. I have the sense this is a pretty fundamental epistemic crux between camps of people making predictions (or suspending judgment!) about AI takeoff.
Yeah, at this point the marginal value add of forecasting/epistemics is in validating/invalidating fundamental assumptions like the software intelligence explosion idea, or the possibility of industrial/chip factory being massively scaled up by AIs, or Moore’s law not ending, rather than on the parameter ranges, because the assumptions overdetermine the conclusion.
Overall, I agree that we were not prioritizing addressing the demographic of skeptics who have detailed reasons for their beliefs. I very much sympathize with disagreeing the framework that others are using to approach takeoff forecasting rather than just parameter settings, I feel similarly to some extent about others’ work (in particular Epoch’s GATE).
However, I disagree that our model assumes a software-driven intelligence explosion. A substantial percentage of our simulations don’t contain such an explosion! You can see that for example the 90th percentile for ASI is >2100. You can totally input your best guesses for the parameters in our model and end up with <50% on a software-driven explosion.
I also think that our scenario and supplements have convinced some skeptics who didn’t have as sophisticated as reasons as e.g. you and Epoch for their disagreements going in. But of course you probably think this is a bad thing so :shrug:
And I’m also worried – as always with this stuff – that there are some people who will look at all those pages and pages of fancy numbers, and think “wow! this sounds crazy but I can’t argue with Serious Expert Research™,” and end up getting convinced even though the document isn’t really trying to convince them in the first place.
Also very much sympathize with this! I’ve had similar concerns about other approaches. I aimed to try to be clear about our levels of uncertainty and how scrappy the model is but perhaps I could have added further caveats? Curious what you would have recommended here.
Finally: I’d be quite interested in your object-level disagreements with our takeoff forecasts. I’ve appreciated your comments on timelines. I’d also be interested in your actual forecasts on timelines/takeoff. Like e.g. your 10/50/90th percentiles, as we gave in our supplements.
FWIW this is the step I disagree with, if I understand what you mean by “try”. See this post.
Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).
In this context, we’re trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it’s not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.[1] More on this here (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).
As the linked post explains, “high precision” here does not mean “the credible interval for the parameter is narrow”. It means that your central/point estimate of the parameter is pinned down to a narrow range, even if you have lots of uncertainty.
It’s notable that you’re just generally arguing against having probabilistic beliefs about events which are unprecedented[1], nothing is specific to this case of doing AI forecasting. You’re mostly objecting to the idea of having (e.g.) medians on events like this.
Of course, the level of precedentedness is continous and from understanding forecasters have successfully done OK at predicting increasingly unprecedented events. Maybe your take is that AI is the most unprecedented event anyone has ever tried to predict. This seems maybe plausible.
arguing against having probabilistic beliefs about events which are unprecedented
Sorry, I’m definitely not saying this. First, in the linked post (see here), I argue that our beliefs should still be probabilistic, just imprecisely so. Second, I’m not drawing a sharp line between “precedented” and “unprecedented.” My point is: Intuitions are only as reliable as the mechanisms that generate them. And given the sparsity of feedback loops[1] and unusual complexity here, I don’t see why the mechanisms generating AGI/ASI forecasting intuitions would be truth-tracking to a high degree of precision. (Cf. Violet Hour’s discussion in Sec. 3 here.)
the level of precedentedness is continous
Right, and that’s consistent with my view. I’m saying, roughly, the degree of imprecision (/width of the interval-valued credence) should increase continuously with the depth of unprecedentedness, among other things.
forecasters have successfully done OK at predicting increasingly unprecedented events
As I note here, our direct evidence only tells us (at best) that people can successfully forecast up to some degree of precision, in some domains. How we ought to extrapolate from this to the case of AGI/ASI forecasting is very underdetermined.
(Yes, I’m aware you meant imprecise probabilities. These aren’t probablities though (in the same sense that a range of numbers isn’t a number), e.g., you’re unwilling to state a median.)
(Replying now bc of the “missed the point” reaction:) To be clear, my concern is that someone without more context might pattern-match the claim “Anthony thinks we shouldn’t have probabilistic beliefs” to “Anthony thinks we have full Knightian uncertainty about everything / doesn’t think we can say any A is more or less likely than any B”. From my experience having discussions about imprecision, conceptual rounding errors are super common, so I think this is a reasonable concern even if you personally find it obvious that “probabilistic” should be read as “using a precise probability distribution”.
FWIW this is the step I disagree with, if I understand what you mean by “try”. See this post.
One way to put this is: what would your preferred state of the timelines discourse be?
In this context, we’re trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it’s not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.[1] More on this here (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).
I totally agree that we can’t pin down the parameters to high precision and disagreement will continue to subsist to a large amount. That’s not a crux for thinking this work is valuable. I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff. I’m not that excited about marginal timelines/takeoff work that doesn’t do this (at least for audiences of AI safety people, I think communicating views to others might still be valuable and I actually view a large part of the timelines forecast in AI 2027 as communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would).
what would your preferred state of the timelines discourse be?
My main recommendation would be, “Don’t pin down probability distributions that are (significantly) more precise than seems justified.” I can’t give an exact set of guidelines for what constitutes “more precise than seems justified” (such is life as a bounded agent!). But to a first approximation:
Suppose I’m doing some modeling, and I find myself thinking, “Hm, what feels like the right median for this? 40? But ehh maybe 50, idk…”
And suppose I can’t point to any particular reason for favoring 40 over 50, or vice versa. (Or, I can point to some reasons for one number but also some reasons for the other, and it’s not clear which are stronger — when I try weighing up these reasons against each other, I find some reasons for one higher-order weighing and some reasons for another, etc. etc.)
This isn’t a problem for every pair of numbers that occurs to us when estimating stuff. If I have to pick between, say, 2030 or 2060 for my AGI timelines median, it seems like I have reason to trust my (imprecise!) intuition[1] that AI progress is going fast enough that 2060 is unreasonable.
Then: I wouldn’t pick just one of 40 or 50 for the median, or just one number in between. I’d include them all.
I totally agree that we can’t pin down the parameters to high precision
I’m not sure I understand your position, then. Do you endorse imprecise probabilities in principle, but report precise distributions for some illustrative purpose? (If so, I’d worry that’s misleading.) My guess is that we’re not yet on the same page about what “pin down the parameters to high precision” means.
I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff
Agreed! I appreciate your detailed transparency in communicating the structure of the model, even if I disagree about the formal epistemology.
communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would
If our beliefs about this domain ought to be significantly imprecise, not just uncertain, then I’d think the more transparent way to communicate your reasoning would be to report an imprecise (yet still quantitative) forecast.
I don’t want to overstate this, tbc. I think this intuition is only trustworthy to the extent that I think it’s a compression of (i) lots of cached understanding I’ve gathered from engaging with timelines research, and (ii) conservative-seeming projections of AI progress that pass enough of a sniff test. If I came into this domain with no prior background, just having a vibe of “2060 is way too far off” wouldn’t be a sufficient justification, I think.
Isn’t it kinda unreasonable to put 10% on superhuman coder in a year if current AIs have a 15 nanosecond time horizon? TBC, it seems fine IMO if the model just isn’t very good at predicting the 10th/90th percentile, especially wiht extreme hyperparameters.
Yeah I guess I think this is probably wrong. Perhaps one way to frame this would be that we would think there was a much lower chance of aggressive superexponentiality if we started at a lower time horizon, so we are sort of hardcoding the superexponentiality parameters to the current time horizon. This might not be the only issue though.
Isn’t it kinda unreasonable to put 10% on superhuman coder in a year if current AIs have a 15 nanosecond time horizon? TBC, it seems fine IMO if the model just isn’t very good at predicting the 10th/90th percentile, especially with extreme hyperparameters.
I also don’t know how they ran this, I tried looking for model code and I couldn’t find it.(Edit: found the code.)I don’t think the defense of being bad at 10% and 90% work because those are defined by the non-super-exponential scenarios, and they are detached from the crux of the problem. Those are the more honest numbers as they actually respond to parameter changes in ways at least attempting to be related to extrapolation.
40-45% of the population, who almost necessarily define the median alone, have a honest, to-infinity singularity at 25-90 months that then gets accelerated a bit further by the secondary super-exponential factor, research progress multipliers. This basically guarantees to a median result in 2-5 years (honestly tighter than that unless you are doing nanoseconds), give or take the additive parameters, that has nothing to do with what people think the model is saying.
If the model is about “how far we are to super-human effort and are extrapolating how long it takes to get there”, a median-defining singularity that is not mentioned except in a table is just not good science.
For another look, here is a 15 millisecond-to-4 hour time horizon comparison just in the super-exponentiated plurality.
This is a prediction with output deviations representing maybe hundredth-SD movements over parameter changes of 10^6 (on the baseline parameter) in an estimation task that includes terms with doubly-exponentiated errors.
I don’t think something like this is defensible based on the justification given in the appendix, which is that forecasters (are we forecasting or or trend-extrapolating pre-super-coders?) think the trend might go straight to infinity before 10 doublings (only 1024x) because humans are better going from 1-2 year tasks than 1-2 month tasks.
Edit: as I am newbie-rate-limited may not be able to talk immediately!
Here is the pure singularity isolated (only the first super-exponential function applied) and the same with normal parameters, which mostly just hide what’s happening:
Thanks a lot for digging into this, appreciate it!
Is the 10% supposed to say 50%?
Yup, very open to the criticism that the possibility and importance of superexponentiality should be highlighted more prominently. Perhaps it should be highlighted in the summary? To be clear, I was not intentionally trying to hide this. For example, it is mentioned but not strongly highlighted in the corresponding expandable in the scenario (“Why we forecast a superhuman coder in early 2027”).
copied from response to Ryan: Yeah I guess I think this is probably wrong. Perhaps one way to frame this would be that we would think there was a much lower chance of aggressive superexponentiality if we started at a lower time horizon, so we are sort of hardcoding the superexponentiality parameters to the current time horizon. This might not be the only issue though.
The superexponential in the simulation is implemented pretty crudely, if you have ideas on how to improve it I’d be interested, I didn’t spend a large amount of time on this. I think this is happening because it’s crudely implemented as each doubling taking 10% shorter, but this already leads to extremely low doubling times by the time the marginal doublings from the starting time horizon are added.
An obvious marginal thing I could do is to add uncertainty as to the shortening factor, though when we tested this and I eyeballed it it didn’t make that big of a difference (but I might have been mistaken). I also think this would ideally be implemented as some chance of going superexponential after each doubling rather than a chance of it happening only from the start, but didn’t have time to implement that.
I will defend the basic dynamics though of substantial probability of a superexponentiality leading to a singularity on an extended version of the METR time horizon dataset being reasonable. And I don’t think it’s obvious whether we model that dynamic as too fast or too slow in general.
Yup, similar reaction to the above graph, I agree the shape looks wrong, interested in suggestions if you have them.
edit: actually I’m pretty confused about this last graph, what code exactly are you running? I don’t think I understand.
Of course, appreciate the response!
This was a rebuttal to Ryan’s defense of the median estimates by saying that, well, the 10/90th might not be as accurate esp with weird parameters, but the 50th might be fine. So I was trying to show that the 50th was the heart of the problem in response.
Not implying intentional hiding! I honestly think that’s pretty irrelevant since the data are purely produced through non-reproducible intuited or verbally justified parameter estimates, so intentionality is completely opaque to readers, which is really the bigger core problem with the exercise. Imagine if climate researchers had made a model where carbon impact on the environment gets twice-to-infinity times larger in 2028 based on “that’s what we think might be happening in Jupiter.” Nobody here would care about intentionality and I don’t think we should care or think it matters most in this case either.
I think this is not true, but I don’t think it is worth litigating separately from the research progress super-exponential factor which is also modeled as already happening. so it is easeir (though I don’t think easy) to defend both individually based on small potatoes evidence of a speedup, but I can’t imagine how both could be defended at once, so that has to come first so we don’t double count evidence.
Here is the parameter setting for the “singularity-only” version:
Then, split out the
calculate_base_timefunction from thecalculate_sc_arrival_yearfunction and plot those results (here is my snippet to adjust the base time to decimal year and limit to the super-exponential group:base_time_in_months[se_mask] / 12 + 2025). You’ll have to do some simple wrangling to expose the se_mask and get the plotting function to work but nothing substantive.Okay, we have a more fundamental disagreement than I previously realized. Can you tell me which step of the following argument you disagree with:
It’s important to forecast AGI timelines.
In order to forecast AGI timelines while taking into account all significantly relevant factors, one needs to use intuitive estimations for several important parameters.
Despite (2), it’s important to try anyways and make the results public to (a) advance the state of public understanding of AGI timelines and (b) make our reasoning for our timelines more transparent than if we gave a vague justification with no model.
Therefore, it’s a good and important thing to make and publish models like ours
A few thoughts:
Climate change forecasting is fundamentally more amenable to grounded quantitative modeling than AGI forecasting, and even there my impression is that there’s substantial disagreement based on qualitative arguments regarding various parameter settings (though I’m very far from an expert on this).
Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).
I’d be curious if you have the same criticism of previous timelines forecasts like Bio Anchors.
I don’t understand your analogy re: Jupiter. In the timelines model, we are trying to predict what will happen in the real world on Earth.
I don’t understand why you think we shouldn’t account for research progress speedups due to AI systems on the way to superhuman coder. And this is clearly a separate dynamic from the trend being possibly superexopnential without these speedups taken into account (which, again, we already see some empirical evidence for this possibility). I’d appreciate if you made object-level arguments against one or both of these factors being included.
Could you share the code on a fork or PR?
I have nothing against publishing models like this! I think the focus should be on modeling assumptions like super-exponentiality and treated as modeling exercises first-and-foremost. Assumptions that highly constrain the model should be first and foremost rather than absent from publicly facing write-ups and only in appendices. Perturbation tests should be conducted and publicly shown. It is true that I don’t think this represents high-quality quantitative work, but that doesn’t mean I don’t think any of it is relevant, even if I think the presentation does not demonstrate a good sense of responsibility. The hype and advertising-ness of the presentation is consistent, and while I have been trying to avoid particular comment thereon, it is a constant weight hanging over the discussion. Publication of a model and breathless speculation are not the same, and I really do not want to litigate the speculative aspects but they are the main thing everyone is talking about including in the New York Times.
I think this has a relatively simple crux
Research progress has empirically slowed according to your own source of EpochAI (Edit: as a percentage of all progress which is the term used in the model. It is constant in absolute terms)
You assume the opposite which would explain away the METR speed-up on its own
So justifying the additional super-exponential term with the METR speed-up is double counting evidence
I can drop my versions of the files in a drive or something, I’m just editing on the fly, so they aren’t intended to be snapshotted, but here you go!
https://drive.google.com/drive/folders/1_e-hnmSy2FoMSD4UiWKWKNKnsapZDm1M?usp=drive_link
Edit:
4. The analogy is that the justification uses analogy to humans which are notably not LLMs. If it is purely based on the METR “speed-up” that should be made more clear as it is a (self-admittedly) weak argument.
(I offered to call with Peter because I’m having trouble understanding his positions over text-based discussion. edit: looks like we will call)
I will briefly say re:
I agree in an ideal world the timelines model would be more rigorous with better structure, more ablation studies, more justifications, etc. However this would have taken a lot longer and my guess is that we would have ended up with similar forecasts in terms of their qualitative conclusions about how plausible AGI in/by 2027 is (though of course, I’m not certain). We didn’t want to let perfect be the enemy of the good/useful (i.e. good/useful in our opinion; seems like the crux may be you disagree that it’s useful).
Apologies if any part of the media stuff seemed like it overclaimed regarding our work.
Looking forward to getting clarity!
“we would have ended up with similar forecasts in terms of their qualitative conclusions about how plausible AGI in/by 2027 is” if this is with regard to perturbation studies of the two super-exponential terms, I believe based on the work I’ve shared in part here it is false, but happy to agree to disagree on the rest :)
I’m curious to hear what conclusions you think we would have came to & should come to. I’m skeptical that they would have been qualitatively different. Perhaps you are going to argue that we shouldn’t put much credence in the superexponential model? What should we put it in instead? Got a better superexponential model for us? Or are you going to say we should stick to exponential?
Thanks for engaging, I’m afraid I can’t join the call due to a schedule conflict but I look forward to hearing about it from Eli!
Ah, I was trying to avoid implying a lack of integrity in the forecasting effort, and as a result ended up implying knowing other things about your mental state.
To restate: I do not think a forecaster not previously committed to achieving a result with a median around that point would have, after doing a perturbation analysis that would display how dominant the two superexponential terms are in predetermining that specific outcome, presented those conclusions without hammering the point that those two superexponential terms are completely determining the topline result and that no other parameters make a meaningful contribution, whether forecasts of compute availability, current capabilities, capabilities required for SC, or otherwise.
Sorry if my previous comment implied something else!
Strongly agree — cf. nostalgebraist’s posts making this point on the bio anchors and AI 2027 models. I have the sense this is a pretty fundamental epistemic crux between camps of people making predictions (or suspending judgment!) about AI takeoff.
Yeah, at this point the marginal value add of forecasting/epistemics is in validating/invalidating fundamental assumptions like the software intelligence explosion idea, or the possibility of industrial/chip factory being massively scaled up by AIs, or Moore’s law not ending, rather than on the parameter ranges, because the assumptions overdetermine the conclusion.
Comment down below:
https://forum.effectivealtruism.org/posts/rv4SJ68pkCQ9BxzpA/?commentId=EgqgffC4F5yZQreCp
I probably agree with you guys more than you realize. You guys might be interested in this DM I sent to nostalgebraist:
FWIW this is the step I disagree with, if I understand what you mean by “try”. See this post.
In this context, we’re trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it’s not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.[1] More on this here (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).
As the linked post explains, “high precision” here does not mean “the credible interval for the parameter is narrow”. It means that your central/point estimate of the parameter is pinned down to a narrow range, even if you have lots of uncertainty.
It’s notable that you’re just generally arguing against having probabilistic beliefs about events which are unprecedented[1], nothing is specific to this case of doing AI forecasting. You’re mostly objecting to the idea of having (e.g.) medians on events like this.
Of course, the level of precedentedness is continous and from understanding forecasters have successfully done OK at predicting increasingly unprecedented events. Maybe your take is that AI is the most unprecedented event anyone has ever tried to predict. This seems maybe plausible.
Sorry, I’m definitely not saying this. First, in the linked post (see here), I argue that our beliefs should still be probabilistic, just imprecisely so. Second, I’m not drawing a sharp line between “precedented” and “unprecedented.” My point is: Intuitions are only as reliable as the mechanisms that generate them. And given the sparsity of feedback loops[1] and unusual complexity here, I don’t see why the mechanisms generating AGI/ASI forecasting intuitions would be truth-tracking to a high degree of precision. (Cf. Violet Hour’s discussion in Sec. 3 here.)
Right, and that’s consistent with my view. I’m saying, roughly, the degree of imprecision (/width of the interval-valued credence) should increase continuously with the depth of unprecedentedness, among other things.
As I note here, our direct evidence only tells us (at best) that people can successfully forecast up to some degree of precision, in some domains. How we ought to extrapolate from this to the case of AGI/ASI forecasting is very underdetermined.
On the actual information of interest (i.e. information about AGI/ASI), that is, not just proxies like forecasting progress in weaker or narrower AI.
(Yes, I’m aware you meant imprecise probabilities. These aren’t probablities though (in the same sense that a range of numbers isn’t a number), e.g., you’re unwilling to state a median.)
(Replying now bc of the “missed the point” reaction:) To be clear, my concern is that someone without more context might pattern-match the claim “Anthony thinks we shouldn’t have probabilistic beliefs” to “Anthony thinks we have full Knightian uncertainty about everything / doesn’t think we can say any A is more or less likely than any B”. From my experience having discussions about imprecision, conceptual rounding errors are super common, so I think this is a reasonable concern even if you personally find it obvious that “probabilistic” should be read as “using a precise probability distribution”.
One way to put this is: what would your preferred state of the timelines discourse be?
I totally agree that we can’t pin down the parameters to high precision and disagreement will continue to subsist to a large amount. That’s not a crux for thinking this work is valuable. I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff. I’m not that excited about marginal timelines/takeoff work that doesn’t do this (at least for audiences of AI safety people, I think communicating views to others might still be valuable and I actually view a large part of the timelines forecast in AI 2027 as communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would).
My main recommendation would be, “Don’t pin down probability distributions that are (significantly) more precise than seems justified.” I can’t give an exact set of guidelines for what constitutes “more precise than seems justified” (such is life as a bounded agent!). But to a first approximation:
Suppose I’m doing some modeling, and I find myself thinking, “Hm, what feels like the right median for this? 40? But ehh maybe 50, idk…”
And suppose I can’t point to any particular reason for favoring 40 over 50, or vice versa. (Or, I can point to some reasons for one number but also some reasons for the other, and it’s not clear which are stronger — when I try weighing up these reasons against each other, I find some reasons for one higher-order weighing and some reasons for another, etc. etc.)
This isn’t a problem for every pair of numbers that occurs to us when estimating stuff. If I have to pick between, say, 2030 or 2060 for my AGI timelines median, it seems like I have reason to trust my (imprecise!) intuition[1] that AI progress is going fast enough that 2060 is unreasonable.
Then: I wouldn’t pick just one of 40 or 50 for the median, or just one number in between. I’d include them all.
I’m not sure I understand your position, then. Do you endorse imprecise probabilities in principle, but report precise distributions for some illustrative purpose? (If so, I’d worry that’s misleading.) My guess is that we’re not yet on the same page about what “pin down the parameters to high precision” means.
Agreed! I appreciate your detailed transparency in communicating the structure of the model, even if I disagree about the formal epistemology.
If our beliefs about this domain ought to be significantly imprecise, not just uncertain, then I’d think the more transparent way to communicate your reasoning would be to report an imprecise (yet still quantitative) forecast.
I don’t want to overstate this, tbc. I think this intuition is only trustworthy to the extent that I think it’s a compression of (i) lots of cached understanding I’ve gathered from engaging with timelines research, and (ii) conservative-seeming projections of AI progress that pass enough of a sniff test. If I came into this domain with no prior background, just having a vibe of “2060 is way too far off” wouldn’t be a sufficient justification, I think.
As a followup: Hopefully this post of mine further clarifies my position, specifically the “Unawareness and superforecasting” section.
Yeah I guess I think this is probably wrong. Perhaps one way to frame this would be that we would think there was a much lower chance of aggressive superexponentiality if we started at a lower time horizon, so we are sort of hardcoding the superexponentiality parameters to the current time horizon. This might not be the only issue though.