Update: After talking to various people, it appears that (contrary to what the poll would suggest) there are at least a few people who answer Question 2 (all three variants) with less than 80%. In light of those conversations, and more thinking on my own, here is my current hot take on how +12 OOMs could turn out to not be enough:
1. Maybe the scaling laws will break. Just because GPT performance has fit a steady line across 5 orders of magnitude so far (or whatever) doesn’t mean it will continue for another 5. Maybe it’ll level off for some reason we don’t yet understand. Arguably this is what happened with LSTMs? Anyhow, for timelines purposes what matters is not whether it’ll level off by the time we are spending +12 OOMs of compute, but rather more like whether it will level off by the time we are spending +6 OOMs of compute. I think it’s rather unlikely to level off that soon, but it might. Maybe 20% chance. If this happens, then probably Amp(GPT-7) and the like wouldn’t work. (80%?) The others are less impacted, but maybe we can assume OmegaStar probably won’t work either. Crystal Nights, SkunkWorks, and Neuromorph… don’t seem to be affected by scaling laws though. If this were the only consideration, my credence would be something like 15% chance that Crystal Nights and OmegaStar don’t work, and then independently, maybe 30% chance that none of the others work too, for a total of 95% answer to Question Two… :/ I could fairly easily be convinced that it’s more like a 40% chance instead of 15% chance, in which case my answer is still something like 85%… :(
2. Maybe the horizon length framework plus scaling laws really will turn out to be a lot more solid than I think. In other words, maybe +12 OOMs is enough to get us some really cool chatbots and whatnot but not anything transformative or PONR-inducing; for those tasks we need long-horizon training… (Medium-horizons can be handled by +12 OOMs). Unsurprisingly to those who’ve read my sequence on takeoff and takeover, I do not think this is very plausible; I’m gonna say something like 10%. (Remember it has to not just apply to standard ML stuff like OmegaStar, but also to amplified GPT-7 and also to Crystal Nights and whatnot. It has to be basically an Iron Law of Learning.) Happily this is independent of point 1 though so that makes for total answer to Q2 of something more like 85%
3. There’s always unknown unknowns. I include “maybe we are data-limited” in this category. Or maybe it turns out that +12 OOMs is enough, and actually +8 OOMs is enough, but we just don’t have the industrial capacity or energy production capacity to scale up nearly that far in the next 20 years or so. I prefer to think of these things as add-ons to the model that shift our timelines back by a couple years, rather than as things that change our answer to Question Two. Unknown unknowns that change our answer to Question Two seem like, well, the thing I mentioned in the text—maybe there’s some super special special sauce that not even Crystal Nights or Neuromorph can find etc. etc. and also Skunkworks turns out to be useless. Yeah… I’m gonna put 5% in this category. Total answer to Question Two is 80% and I’m feeling pretty reasonable about it.
4. There’s biases. Some people I talked to basically said “Yeah, 80%+ seems right to me too, but I think we should correct for biases and assume everything is more difficult than it appears; if it seems like it’s very likely enough to us, that means it’s 50% likely enough.” I don’t currently endorse this, because I think that the biases pushing in the opposite direction—biases of respectability, anti-weirdness, optimism, etc.--are probably on the whole stronger. Also, the people in the past who used human brain milestone to forecast AI seem to have been surprisingly right, of course it’s too early to say but reality really is looking exactly like it should look if they were right...
5. There’s deference to the opinions of others, e.g. AI scientists in academia, economists forecasting GWP trends, the financial markets… My general response is “fuck that.” If you are interested I can say more about why I feel this way; ultimately I do in fact make a mild longer-timelines update as a result of this but I do so grudgingly. Also, Roodman’s model actually predicts 2037, not 2047. And that’s not even taking into account how AI-PONR will probably be a few years beforehand!
So all in all my credence has gone down from 90% to 80% for Question Two, but I’ve also become more confident that I’m basically correct, that I’m not the crazy one here. Because now I understand the arguments people gave, the models people have, for why the number might be less than 80%, and I have evaluated them and they don’t seem that strong.
I’d love to hear more thoughts and takes by the way, if you have any please comment!
It is irrelevant to this post, because this post is about what our probability distribution over orders of magnitude of compute should be like. Once we have said distribution, then we can ask: How quickly (in clock time) will we progress through the distribution / explore more OOMs of compute? Then the AI and compute trend, and the update to it, become relevant.
But not super relevant IMO. The AI and Compute trend was way too fast to be sustained, people at the time even said so. This recent halt in the trend is not surprising. What matters is what the trend will look like going forward, e.g. over the next 10 years, over the next 20 years, etc.
It can be broken down into two components: Cost reduction and spending increase.
Ajeya separately estimates each component for the near term (5 years) and for the long-term trend beyond.
I mostly defer to her judgment on this, with large uncertainty. (Ajeya thinks costs will halve every 2.5 years, which is slower than the 1.5 years average throughout all history, but justifiable given how Moore’s Law is said to be dying now. As for spending increases, she thinks it will take decades to ramp up to trillion-dollar expenditures, whereas I am more uncertain and think it could maybe happen by 2030 idk.)
I feel quite confident in the following claim: Conditional on +6 OOMs being enough with 2020′s ideas, it’ll happen by 2030. Indeed, conditional on +8 OOMs being enough with 2020′s ideas, I think it’ll probably happen by 2030. If you are interested in more of my arguments for this stuff, I have some slides I could share slash I’d love to chat with you about this! :)
If the AI and compute trend is just a blip, then doesn’t that return us to the previous trend line in the graph you show at the beginning, where we progress about 2 ooms a decade? (More accurately, 1 oom every 6-7 years, or, 8 ooms in 5 decades.)
Ignoring AI and compute, then: if we believe +12 ooms in 2016 means great danger in 2020, we should believe that roughly 75 years after 2016, we are at most four years from the danger zone.
Whereas, if we extrapolate the AI-and-compute trend, +12 ooms is like jumping 12 years in the future; so the idea of risk by 2030 makes sense.
So I don’t get how your conclusion can be so independent of AI-and-compute.
Sorry, somehow I missed this. Basically, the answer is that we definitely shouldn’t just extrapolate out the AI and compute trend into the future, and Ajeya’s and my predictions are not doing that. Instead we are assuming something more like the historic 2 ooms a decade trend, combined with some amount of increased spending conditional on us being close to AGI/TAI/etc. Hence my conditional claim above:
Conditional on +6 OOMs being enough with 2020′s ideas, it’ll happen by 2030. Indeed, conditional on +8 OOMs being enough with 2020′s ideas, I think it’ll probably happen by 2030.
If you want to discuss this more with me, I’d love to, how bout we book a call?
Update: After talking to various people, it appears that (contrary to what the poll would suggest) there are at least a few people who answer Question 2 (all three variants) with less than 80%. In light of those conversations, and more thinking on my own, here is my current hot take on how +12 OOMs could turn out to not be enough:
1. Maybe the scaling laws will break. Just because GPT performance has fit a steady line across 5 orders of magnitude so far (or whatever) doesn’t mean it will continue for another 5. Maybe it’ll level off for some reason we don’t yet understand. Arguably this is what happened with LSTMs? Anyhow, for timelines purposes what matters is not whether it’ll level off by the time we are spending +12 OOMs of compute, but rather more like whether it will level off by the time we are spending +6 OOMs of compute. I think it’s rather unlikely to level off that soon, but it might. Maybe 20% chance. If this happens, then probably Amp(GPT-7) and the like wouldn’t work. (80%?) The others are less impacted, but maybe we can assume OmegaStar probably won’t work either. Crystal Nights, SkunkWorks, and Neuromorph… don’t seem to be affected by scaling laws though. If this were the only consideration, my credence would be something like 15% chance that Crystal Nights and OmegaStar don’t work, and then independently, maybe 30% chance that none of the others work too, for a total of 95% answer to Question Two… :/ I could fairly easily be convinced that it’s more like a 40% chance instead of 15% chance, in which case my answer is still something like 85%… :(
2. Maybe the horizon length framework plus scaling laws really will turn out to be a lot more solid than I think. In other words, maybe +12 OOMs is enough to get us some really cool chatbots and whatnot but not anything transformative or PONR-inducing; for those tasks we need long-horizon training… (Medium-horizons can be handled by +12 OOMs). Unsurprisingly to those who’ve read my sequence on takeoff and takeover, I do not think this is very plausible; I’m gonna say something like 10%. (Remember it has to not just apply to standard ML stuff like OmegaStar, but also to amplified GPT-7 and also to Crystal Nights and whatnot. It has to be basically an Iron Law of Learning.) Happily this is independent of point 1 though so that makes for total answer to Q2 of something more like 85%
3. There’s always unknown unknowns. I include “maybe we are data-limited” in this category. Or maybe it turns out that +12 OOMs is enough, and actually +8 OOMs is enough, but we just don’t have the industrial capacity or energy production capacity to scale up nearly that far in the next 20 years or so. I prefer to think of these things as add-ons to the model that shift our timelines back by a couple years, rather than as things that change our answer to Question Two. Unknown unknowns that change our answer to Question Two seem like, well, the thing I mentioned in the text—maybe there’s some super special special sauce that not even Crystal Nights or Neuromorph can find etc. etc. and also Skunkworks turns out to be useless. Yeah… I’m gonna put 5% in this category. Total answer to Question Two is 80% and I’m feeling pretty reasonable about it.
4. There’s biases. Some people I talked to basically said “Yeah, 80%+ seems right to me too, but I think we should correct for biases and assume everything is more difficult than it appears; if it seems like it’s very likely enough to us, that means it’s 50% likely enough.” I don’t currently endorse this, because I think that the biases pushing in the opposite direction—biases of respectability, anti-weirdness, optimism, etc.--are probably on the whole stronger. Also, the people in the past who used human brain milestone to forecast AI seem to have been surprisingly right, of course it’s too early to say but reality really is looking exactly like it should look if they were right...
5. There’s deference to the opinions of others, e.g. AI scientists in academia, economists forecasting GWP trends, the financial markets… My general response is “fuck that.” If you are interested I can say more about why I feel this way; ultimately I do in fact make a mild longer-timelines update as a result of this but I do so grudgingly. Also, Roodman’s model actually predicts 2037, not 2047. And that’s not even taking into account how AI-PONR will probably be a few years beforehand!
So all in all my credence has gone down from 90% to 80% for Question Two, but I’ve also become more confident that I’m basically correct, that I’m not the crazy one here. Because now I understand the arguments people gave, the models people have, for why the number might be less than 80%, and I have evaluated them and they don’t seem that strong.
I’d love to hear more thoughts and takes by the way, if you have any please comment!
So, how does the update to the AI and compute trend factor in?
It is irrelevant to this post, because this post is about what our probability distribution over orders of magnitude of compute should be like. Once we have said distribution, then we can ask: How quickly (in clock time) will we progress through the distribution / explore more OOMs of compute? Then the AI and compute trend, and the update to it, become relevant.
But not super relevant IMO. The AI and Compute trend was way too fast to be sustained, people at the time even said so. This recent halt in the trend is not surprising. What matters is what the trend will look like going forward, e.g. over the next 10 years, over the next 20 years, etc.
It can be broken down into two components: Cost reduction and spending increase.
Ajeya separately estimates each component for the near term (5 years) and for the long-term trend beyond.
I mostly defer to her judgment on this, with large uncertainty. (Ajeya thinks costs will halve every 2.5 years, which is slower than the 1.5 years average throughout all history, but justifiable given how Moore’s Law is said to be dying now. As for spending increases, she thinks it will take decades to ramp up to trillion-dollar expenditures, whereas I am more uncertain and think it could maybe happen by 2030 idk.)
I feel quite confident in the following claim: Conditional on +6 OOMs being enough with 2020′s ideas, it’ll happen by 2030. Indeed, conditional on +8 OOMs being enough with 2020′s ideas, I think it’ll probably happen by 2030. If you are interested in more of my arguments for this stuff, I have some slides I could share slash I’d love to chat with you about this! :)
If the AI and compute trend is just a blip, then doesn’t that return us to the previous trend line in the graph you show at the beginning, where we progress about 2 ooms a decade? (More accurately, 1 oom every 6-7 years, or, 8 ooms in 5 decades.)
Ignoring AI and compute, then: if we believe +12 ooms in 2016 means great danger in 2020, we should believe that roughly 75 years after 2016, we are at most four years from the danger zone.
Whereas, if we extrapolate the AI-and-compute trend, +12 ooms is like jumping 12 years in the future; so the idea of risk by 2030 makes sense.
So I don’t get how your conclusion can be so independent of AI-and-compute.
Sorry, somehow I missed this. Basically, the answer is that we definitely shouldn’t just extrapolate out the AI and compute trend into the future, and Ajeya’s and my predictions are not doing that. Instead we are assuming something more like the historic 2 ooms a decade trend, combined with some amount of increased spending conditional on us being close to AGI/TAI/etc. Hence my conditional claim above:
If you want to discuss this more with me, I’d love to, how bout we book a call?
Is there a reference for this?
https://www.gwern.net/images/ai/gpt/2020-kaplan-figure7-rnnsvstransformers.png
What Gwern said. :) But I don’t know for sure what the person I talked to had in mind.