FWIW this is the step I disagree with, if I understand what you mean by “try”. See this post.
Forecasts which include intuitive estimations are commonplace and often useful (see e.g. intelligence analysis, Superforecasting, prediction markets, etc.).
In this context, we’re trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it’s not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.[1] More on this here (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).
As the linked post explains, “high precision” here does not mean “the credible interval for the parameter is narrow”. It means that your central/point estimate of the parameter is pinned down to a narrow range, even if you have lots of uncertainty.
It’s notable that you’re just generally arguing against having probabilistic beliefs about events which are unprecedented[1], nothing is specific to this case of doing AI forecasting. You’re mostly objecting to the idea of having (e.g.) medians on events like this.
Of course, the level of precedentedness is continous and from understanding forecasters have successfully done OK at predicting increasingly unprecedented events. Maybe your take is that AI is the most unprecedented event anyone has ever tried to predict. This seems maybe plausible.
arguing against having probabilistic beliefs about events which are unprecedented
Sorry, I’m definitely not saying this. First, in the linked post (see here), I argue that our beliefs should still be probabilistic, just imprecisely so. Second, I’m not drawing a sharp line between “precedented” and “unprecedented.” My point is: Intuitions are only as reliable as the mechanisms that generate them. And given the sparsity of feedback loops[1] and unusual complexity here, I don’t see why the mechanisms generating AGI/ASI forecasting intuitions would be truth-tracking to a high degree of precision. (Cf. Violet Hour’s discussion in Sec. 3 here.)
the level of precedentedness is continous
Right, and that’s consistent with my view. I’m saying, roughly, the degree of imprecision (/width of the interval-valued credence) should increase continuously with the depth of unprecedentedness, among other things.
forecasters have successfully done OK at predicting increasingly unprecedented events
As I note here, our direct evidence only tells us (at best) that people can successfully forecast up to some degree of precision, in some domains. How we ought to extrapolate from this to the case of AGI/ASI forecasting is very underdetermined.
(Yes, I’m aware you meant imprecise probabilities. These aren’t probablities though (in the same sense that a range of numbers isn’t a number), e.g., you’re unwilling to state a median.)
(Replying now bc of the “missed the point” reaction:) To be clear, my concern is that someone without more context might pattern-match the claim “Anthony thinks we shouldn’t have probabilistic beliefs” to “Anthony thinks we have full Knightian uncertainty about everything / doesn’t think we can say any A is more or less likely than any B”. From my experience having discussions about imprecision, conceptual rounding errors are super common, so I think this is a reasonable concern even if you personally find it obvious that “probabilistic” should be read as “using a precise probability distribution”.
FWIW this is the step I disagree with, if I understand what you mean by “try”. See this post.
One way to put this is: what would your preferred state of the timelines discourse be?
In this context, we’re trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it’s not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.[1] More on this here (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).
I totally agree that we can’t pin down the parameters to high precision and disagreement will continue to subsist to a large amount. That’s not a crux for thinking this work is valuable. I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff. I’m not that excited about marginal timelines/takeoff work that doesn’t do this (at least for audiences of AI safety people, I think communicating views to others might still be valuable and I actually view a large part of the timelines forecast in AI 2027 as communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would).
what would your preferred state of the timelines discourse be?
My main recommendation would be, “Don’t pin down probability distributions that are (significantly) more precise than seems justified.” I can’t give an exact set of guidelines for what constitutes “more precise than seems justified” (such is life as a bounded agent!). But to a first approximation:
Suppose I’m doing some modeling, and I find myself thinking, “Hm, what feels like the right median for this? 40? But ehh maybe 50, idk…”
And suppose I can’t point to any particular reason for favoring 40 over 50, or vice versa. (Or, I can point to some reasons for one number but also some reasons for the other, and it’s not clear which are stronger — when I try weighing up these reasons against each other, I find some reasons for one higher-order weighing and some reasons for another, etc. etc.)
This isn’t a problem for every pair of numbers that occurs to us when estimating stuff. If I have to pick between, say, 2030 or 2060 for my AGI timelines median, it seems like I have reason to trust my (imprecise!) intuition[1] that AI progress is going fast enough that 2060 is unreasonable.
Then: I wouldn’t pick just one of 40 or 50 for the median, or just one number in between. I’d include them all.
I totally agree that we can’t pin down the parameters to high precision
I’m not sure I understand your position, then. Do you endorse imprecise probabilities in principle, but report precise distributions for some illustrative purpose? (If so, I’d worry that’s misleading.) My guess is that we’re not yet on the same page about what “pin down the parameters to high precision” means.
I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff
Agreed! I appreciate your detailed transparency in communicating the structure of the model, even if I disagree about the formal epistemology.
communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would
If our beliefs about this domain ought to be significantly imprecise, not just uncertain, then I’d think the more transparent way to communicate your reasoning would be to report an imprecise (yet still quantitative) forecast.
I don’t want to overstate this, tbc. I think this intuition is only trustworthy to the extent that I think it’s a compression of (i) lots of cached understanding I’ve gathered from engaging with timelines research, and (ii) conservative-seeming projections of AI progress that pass enough of a sniff test. If I came into this domain with no prior background, just having a vibe of “2060 is way too far off” wouldn’t be a sufficient justification, I think.
FWIW this is the step I disagree with, if I understand what you mean by “try”. See this post.
In this context, we’re trying to forecast radically unprecedented events, occurring on long subjective time horizons, where we have little reason to expect these intuitive estimates to be honed by empirical feedback. Peer disagreement is also unusually persistent in this domain. So it’s not at all obvious to me that, based on superforecasting track records, we can trust that our intuitions pin down these parameters to a sufficient degree of precision.[1] More on this here (this is not a comprehensive argument for my view, tbc; hoping to post something spelling this out more soon-ish!).
As the linked post explains, “high precision” here does not mean “the credible interval for the parameter is narrow”. It means that your central/point estimate of the parameter is pinned down to a narrow range, even if you have lots of uncertainty.
It’s notable that you’re just generally arguing against having probabilistic beliefs about events which are unprecedented[1], nothing is specific to this case of doing AI forecasting. You’re mostly objecting to the idea of having (e.g.) medians on events like this.
Of course, the level of precedentedness is continous and from understanding forecasters have successfully done OK at predicting increasingly unprecedented events. Maybe your take is that AI is the most unprecedented event anyone has ever tried to predict. This seems maybe plausible.
Sorry, I’m definitely not saying this. First, in the linked post (see here), I argue that our beliefs should still be probabilistic, just imprecisely so. Second, I’m not drawing a sharp line between “precedented” and “unprecedented.” My point is: Intuitions are only as reliable as the mechanisms that generate them. And given the sparsity of feedback loops[1] and unusual complexity here, I don’t see why the mechanisms generating AGI/ASI forecasting intuitions would be truth-tracking to a high degree of precision. (Cf. Violet Hour’s discussion in Sec. 3 here.)
Right, and that’s consistent with my view. I’m saying, roughly, the degree of imprecision (/width of the interval-valued credence) should increase continuously with the depth of unprecedentedness, among other things.
As I note here, our direct evidence only tells us (at best) that people can successfully forecast up to some degree of precision, in some domains. How we ought to extrapolate from this to the case of AGI/ASI forecasting is very underdetermined.
On the actual information of interest (i.e. information about AGI/ASI), that is, not just proxies like forecasting progress in weaker or narrower AI.
(Yes, I’m aware you meant imprecise probabilities. These aren’t probablities though (in the same sense that a range of numbers isn’t a number), e.g., you’re unwilling to state a median.)
(Replying now bc of the “missed the point” reaction:) To be clear, my concern is that someone without more context might pattern-match the claim “Anthony thinks we shouldn’t have probabilistic beliefs” to “Anthony thinks we have full Knightian uncertainty about everything / doesn’t think we can say any A is more or less likely than any B”. From my experience having discussions about imprecision, conceptual rounding errors are super common, so I think this is a reasonable concern even if you personally find it obvious that “probabilistic” should be read as “using a precise probability distribution”.
One way to put this is: what would your preferred state of the timelines discourse be?
I totally agree that we can’t pin down the parameters to high precision and disagreement will continue to subsist to a large amount. That’s not a crux for thinking this work is valuable. I think this sort of work is valuable because it introduces new, comprehensive-ish frameworks for thinking about timelines/takeoff. I’m not that excited about marginal timelines/takeoff work that doesn’t do this (at least for audiences of AI safety people, I think communicating views to others might still be valuable and I actually view a large part of the timelines forecast in AI 2027 as communicating the reasoning behind our beliefs in a more transparent way than a non-quantitative approach would).
My main recommendation would be, “Don’t pin down probability distributions that are (significantly) more precise than seems justified.” I can’t give an exact set of guidelines for what constitutes “more precise than seems justified” (such is life as a bounded agent!). But to a first approximation:
Suppose I’m doing some modeling, and I find myself thinking, “Hm, what feels like the right median for this? 40? But ehh maybe 50, idk…”
And suppose I can’t point to any particular reason for favoring 40 over 50, or vice versa. (Or, I can point to some reasons for one number but also some reasons for the other, and it’s not clear which are stronger — when I try weighing up these reasons against each other, I find some reasons for one higher-order weighing and some reasons for another, etc. etc.)
This isn’t a problem for every pair of numbers that occurs to us when estimating stuff. If I have to pick between, say, 2030 or 2060 for my AGI timelines median, it seems like I have reason to trust my (imprecise!) intuition[1] that AI progress is going fast enough that 2060 is unreasonable.
Then: I wouldn’t pick just one of 40 or 50 for the median, or just one number in between. I’d include them all.
I’m not sure I understand your position, then. Do you endorse imprecise probabilities in principle, but report precise distributions for some illustrative purpose? (If so, I’d worry that’s misleading.) My guess is that we’re not yet on the same page about what “pin down the parameters to high precision” means.
Agreed! I appreciate your detailed transparency in communicating the structure of the model, even if I disagree about the formal epistemology.
If our beliefs about this domain ought to be significantly imprecise, not just uncertain, then I’d think the more transparent way to communicate your reasoning would be to report an imprecise (yet still quantitative) forecast.
I don’t want to overstate this, tbc. I think this intuition is only trustworthy to the extent that I think it’s a compression of (i) lots of cached understanding I’ve gathered from engaging with timelines research, and (ii) conservative-seeming projections of AI progress that pass enough of a sniff test. If I came into this domain with no prior background, just having a vibe of “2060 is way too far off” wouldn’t be a sufficient justification, I think.
As a followup: Hopefully this post of mine further clarifies my position, specifically the “Unawareness and superforecasting” section.