Suppose your view was that P(AGI if no pause/slow before 2050) = 80%. Then, if we condition on AGI after 2050, surely most of the probability mass isn’t due to pausing/slowing right?
So, what would be the mechanism if not some sort of technical research or exogenous factor (e.g. society getting wiser) over the intervening time.
Note that the full quote in context is:
There’s a Decent Chance of Having Decades
In a similar vein as the above, nobody associated with AI 2027 (or the market, or me) think there’s more than a 95% chance that transformative AI will happen in the next twenty years! I think most of the authors probably think there’s significantly less than a 90% chance of transformative superintelligence before 2045.
Daniel Kokotajlo expressed on the Win-Win podcast (I think) that he is much less doomy about the prospect of things going well if superintelligence is developed after 2030 than before 2030, and I agree. I think if we somehow make it to 2050 without having handed the planet over to AI (or otherwise causing a huge disaster), we’re pretty likely to be in the clear. And, according to everyone involved, that is plausible (but unlikely).
If this is how we win, we should be pushing hard to make it happen and slow things down!
I’m confused by this. I think you’re not using “condition on” in the technical sense, but instead asking how P(AGI > 2050) is effected by P(pause). Is that right?
Assuming it is, we can write: P(AGI>2050) = 100% - P(AGI<2050) = 100% - P(AGI<2050|pause)P(pause) - P(AGI<2050|no pause)P(no pause)
To make this an expression of just P(pause) we need to assume some values of both P(AGI<2050|no pause) and P(AGI<2050|pause).
You suggested P(AGI<2050|no pause) := 80%. Let’s also say that P(AGI<2050|pause) := 30%.
In other words, with these parameters, a majority of the probability mass for making it to 2050 without AGI comes from a pause if the probability of pausing is above 40%.
(Note: 80% is too low for me, personally. I’m at closer to 95% on P(AGI < 2050|no pause). (Edit: oh wait, does “AGI” mean “transformative superintelligence”?) This would make the equation 5% + 65% P(pause), meaning the majority of mass comes from pausing if the probability of pausing is above 1⁄7. Whether 30% is the right number depends heavily on what “pause” means. Arguably it should be 0%, which would make the equation 5% + 95% P(pause), and the relevant threshold 1/19th.)
But actually I think I’m probably confused about what you’re trying to express. Can you say more words?
Not that important to get into, but I’d guess the probability of >3 decade long coordinated pause prior to “very scary AI (whatever you were worried might takeover)” (which is maybe better to talk about than AGI) is like 3% and I’m sympathetic to lower.
I’m skeptical of 95% on P(AGI < 2050|no pause) unless you mean something much weaker than “AI which can automate virtually all cognitive work” when you say AGI. Seems too confident for this type of event IMO.
Cool. Your definition of AGI seems reasonable. Sounds like we probably disagree about confidence and timelines. (My confidence, I believe, matches Metaculus. [Edit: It doesn’t! I’m embarrassed to have claimed this.])
I agree that we seem not to be on the path of pausing. Is your argument “because pausing is extremely unlikely per se, most of the timelines where we make it to 2050 don’t have a pause”? If one assumes that we won’t pause, I agree that the majority of probability mass for X doesn’t involve a pause, for all X, including making it to 2050.
I generally don’t think it’s a good idea to put a probability on things where you have a significant ability to decide the outcome (i.e. probability of getting divorced), and instead encourage you to believe in pausing.
You are right and I am wrong. Oops. After writing my comment I scrolled up to the top of my post, saw the graph from Manafold (not Metaculus), thought “huh, I forgot the market was so confident” and edited in my parenthetical without thinking. This is even more embarrassing because no market question is actually about the probability conditional on no pause occurring, which is a potentially important factor. I definitely shouldn’t have added that text. Thank you.
(I will point out, as a bit of an aside, that economically transformative AI seems like a different threshold than AGI. My sense is that if an AGI takes a million dollars an hour to run an instance, it’s still an AGI, but it won’t be economically transformative unless it’s substantially superintelligent or becomes much cheaper.
I generally don’t think it’s a good idea to put a probability on things where you have a significant ability to decide the outcome (i.e. probability of getting divorced), and instead encourage you to believe in pausing.
In this case, I can at least talk about the probability of a multi decade pause (with the motivation of delaying AI etc) if I were to be hit by a bus tomorrow. My number is unchanged, around 3%. (Maybe there are some good arguments for higher, I’m not sure.)
I agree that if everyone in my decision-theoretic reference class stopped trying to pause AI (perhaps because of being hit by buses), the chance of a pause is near 0.
Suppose your view was that P(AGI if no pause/slow before 2050) = 80%. Then, if we condition on AGI after 2050, surely most of the probability mass isn’t due to pausing/slowing right?
So, what would be the mechanism if not some sort of technical research or exogenous factor (e.g. society getting wiser) over the intervening time.
Note that the full quote in context is:
I’m confused by this. I think you’re not using “condition on” in the technical sense, but instead asking how P(AGI > 2050) is effected by P(pause). Is that right?
Assuming it is, we can write:
P(AGI>2050) = 100% - P(AGI<2050) = 100% - P(AGI<2050|pause)P(pause) - P(AGI<2050|no pause)P(no pause)
To make this an expression of just P(pause) we need to assume some values of both P(AGI<2050|no pause) and P(AGI<2050|pause).
You suggested P(AGI<2050|no pause) := 80%. Let’s also say that P(AGI<2050|pause) := 30%.
P(AGI>2050) = 100% - (30%)P(pause) - (80%)(100% - P(pause))
= 100% − 30%P − 80% + 80%P = 20% + 50% P(pause)
In other words, with these parameters, a majority of the probability mass for making it to 2050 without AGI comes from a pause if the probability of pausing is above 40%.
(Note: 80% is too low for me, personally. I’m at closer to 95% on P(AGI < 2050|no pause). (Edit: oh wait, does “AGI” mean “transformative superintelligence”?) This would make the equation 5% + 65% P(pause), meaning the majority of mass comes from pausing if the probability of pausing is above 1⁄7. Whether 30% is the right number depends heavily on what “pause” means. Arguably it should be 0%, which would make the equation 5% + 95% P(pause), and the relevant threshold 1/19th.)
But actually I think I’m probably confused about what you’re trying to express. Can you say more words?
Not that important to get into, but I’d guess the probability of >3 decade long coordinated pause prior to “very scary AI (whatever you were worried might takeover)” (which is maybe better to talk about than AGI) is like 3% and I’m sympathetic to lower.
I’m skeptical of 95% on P(AGI < 2050|no pause) unless you mean something much weaker than “AI which can automate virtually all cognitive work” when you say AGI. Seems too confident for this type of event IMO.
Cool. Your definition of AGI seems reasonable. Sounds like we probably disagree about confidence and timelines. (My confidence, I believe, matches Metaculus. [Edit: It doesn’t! I’m embarrassed to have claimed this.])
I agree that we seem not to be on the path of pausing. Is your argument “because pausing is extremely unlikely per se, most of the timelines where we make it to 2050 don’t have a pause”? If one assumes that we won’t pause, I agree that the majority of probability mass for X doesn’t involve a pause, for all X, including making it to 2050.
I generally don’t think it’s a good idea to put a probability on things where you have a significant ability to decide the outcome (i.e. probability of getting divorced), and instead encourage you to believe in pausing.
I don’t think Metaculus is that confident. Some questions:
https://www.metaculus.com/questions/19356/transformative-ai-date/
https://www.metaculus.com/questions/5406/world-output-doubles-in-4-years-by-2050/
https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/ (the resolution criteria for this is much weaker than AGI and reasonably likely to trigger much earlier).
Even the last of these has only ~80% by 2050.
You are right and I am wrong. Oops. After writing my comment I scrolled up to the top of my post, saw the graph from Manafold (not Metaculus), thought “huh, I forgot the market was so confident” and edited in my parenthetical without thinking. This is even more embarrassing because no market question is actually about the probability conditional on no pause occurring, which is a potentially important factor. I definitely shouldn’t have added that text. Thank you.
(I will point out, as a bit of an aside, that economically transformative AI seems like a different threshold than AGI. My sense is that if an AGI takes a million dollars an hour to run an instance, it’s still an AGI, but it won’t be economically transformative unless it’s substantially superintelligent or becomes much cheaper.
Still, I take my lumps.)
In this case, I can at least talk about the probability of a multi decade pause (with the motivation of delaying AI etc) if I were to be hit by a bus tomorrow. My number is unchanged, around 3%. (Maybe there are some good arguments for higher, I’m not sure.)
I agree that if everyone in my decision-theoretic reference class stopped trying to pause AI (perhaps because of being hit by buses), the chance of a pause is near 0.