I think of AGI (and human-level intelligence) as the cloud, and superintelligence as being above the cloud. They are useful concepts, despite their vagueness. But they’re markedly less useful when you get close to them.
So I actually like the cloud metaphor. But as you say
Regarding AGI, it’s already getting a bit misty. In February there was a piece in Nature arguing that the current level of frontier AI should count as AGI. I’d set the bar a bit higher than that, but I agree it is already debatable whether we’re in the cloud.
it’s debatable whether we’re in the cloud, which means that perhaps we should be forecasting other things than “AGI”.
And you do give some more specific milestones, like
For my purposes, I think the key threshold is when the system is capable enough that there are dramatic changes to the world — civilisational changes. For example, the point where AI could take over from humanity were it misaligned, or it has made 50% of people permanently unemployable, or has doubled the global rate of technological progress.
but (1) “the point where AI could take over from humanity were it misaligned”, (2) “[the point where] it has made 50% of people permanently unemployable”, and (3) “[the point where it] has doubled the global rate of technological progress” seem to me plausibly quite distinct. I expect [ETA[1]: find it plausible for] (1) to come years after (2) and (3); and (3) probably before (2), but I have substantial uncertainty here. Maybe you could get (2) before (3) if you really crank boring LLM training the right way + robotics. And maybe powerful optimization is out there to screw us, and (1) comes before (2) and (3). If I were to forecast it in a principled probabilistic way, I would end up with three distinct, but somewhat dependent probability distributions.
Here’s a nice graph 80,000 Hours put together of how the average forecasted time until AGI on the Metaculus prediction site has shortened from about 50 years to about 5 years in just a 5-year window:
(1) “the point where AI could take over from humanity were it misaligned”, (2) “[the point where] it has made 50% of people permanently unemployable”, and (3) “[the point where it] has doubled the global rate of technological progress” seem to me plausibly quite distinct. I expect (1) to come years after (2) and (3)
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster,[1] a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI[2]
Conclusion. if you somehow don’t have wildly superhuman AI at the beginning of this thought experiment, you will have it in at most .
(On the surface, it looks like I’m saying “if P then Q”, where P is (2) AND (3) AND not-(1) (using your numbering of events), and Q is (1)-soon, so making a claim about the conditional in which P is true. But that’s not really what I mean to say. Like, to me P is true in some sorts of weird magic worlds carrying (let’s say) probability in which I wouldn’t actually have a strong sense of whether Q. But I’m not really making a claim about the conditional, I’m more trying to argue that P AND not-Q doesn’t hang together (which could happen because P itself already doesn’t hang together, and indeed I think that). That is, it’s really an argument by contradiction that by default (1) doesn’t come much later than (2)+(3), not a claim about the conditional.)
Another pathway that I have substantial mass on: [the first learning setup in which novel mental structure creation / novel domain learning starts to work basically at all] produces a wildly superhuman AI. Like, stuff gets crazy inside a single training/growth process.
the pace of conceptual work on AI algorithms is like >100x faster
In such a case I expect these AI researchers to pick all the low- and medium-hanging fruit at the then-current compute level/hardware technology, and then the algorithmic progress gets saturated until new-gen chips are produced in quality. Check this: https://www.lesswrong.com/posts/sGNFtWbXiLJg2hLzK
Since my Claim 1 is about the conceptual work input being 100x sped up, not some final output being 100x sped up, I’ll take you to be disagreeing with Claim 2. So the question is: is 10 years of thinking about AI algorithms followed by 1 month of retraining sufficient [to get from AI that causes of people to be permanently unemployable to crazy smart AI]? In other words, if one is only going to be able to pick low-hanging and medium-hanging fruit in 10 years, is picking those sufficient to get to crazy smart AI from that point? I claim that the answer is yes; some quick points:
I think we should imagine the fruits at the beginning of this to not have been well-picked (supposing a crazy smart AI does not already exist).
Trusting Byrnes’s decomposition of the 7 year 600x nanochat cost improvement, that’s 6x from hardware and 100x from non-hardware. That would give some sort of baseline guess of for 10 years. Ok, but maybe we should apply some adjustments to the factors. In particular, what about data? On the one hand, it will be tough to collect a lot of data from humans quickly in our scenario. On the other hand, it will be very easy to collect [a lot more data [than we have from humans]] from AIs in this scenario, and by that point this will probably be overall better. On the first hand again, maybe we should imagine data not mattering so much at that point. On the second hand again, all things considered that’s actually conceptually correlated with fooming far past human level quickly. We should also apply some global adjustment down for having less time for experiments to run.
Byrnes explicitly does not include algo ideas that “are not about doing the same thing more efficiently, but rather about trying to do something different instead”. See Section 1.5 of his post. But these clearly should be included in our context here, and are majorly important imo. E.g., curating curricula, creating problems for oneself in a different way, coming up with good ways to reward problem-creation, creating more nested levels of problem-solving with their own rewards, coming up with other ways to make rewards denser / track progress better, creating tools for oneself, various IDA ideas (beyond those already mentioned), etc.. There are also various ways humans get smarter over centuries and over a lifetime that should also count for our purposes as “algo progress” if the AIs can carry them out, e.g. inventing+acquiring new concepts, questions, methods, and skills, and just knowing more.
In our scenario, coming up with an arbitrarily different new AI design is also legitimate, as long as this AI can be created/trained/grown in at most 1 month.
Tbh a lot of my belief that you get a lot of progress just comes from it being an extremely high-dimensional design space and there surely being lots of things one can do so much better in there.
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster, a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
Unless humans are strictly needed to orchestrate the AIs, but a world where they have thinking coherent enough to make massive algorithmic progress, but incoherent enough to pursue this competently, seems super weird, but, hey, maybe Moravec will come to bite us again!
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI
You mean something like serial, uninterrupted, focused thinking, like a WBE of a very-high-g AI researcher that doesn’t need sleep?
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
this makes me want to ask: are you tracking the difference between the event “50 of current human jobs are basically automated” and the event “50 of humans are such that it basically does not make sense to employ them”. like, the former has probably happened multiple times in history, whereas the latter is unprecedented. what you’re saying makes more sense to me if you have the former in mind, but we’re talking about the latter (“people being permanently unemployable”). i have significant probability that you are tracking this correctly already but wanted to check just in case
(I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.)
(to make sure we’re on the same page: in my view, this is unlikely to be a somewhat stable situation)
this makes me want to ask: are you tracking the difference between the event “50 of current human jobs are basically automated” and the event “50 of humans are such that it basically does not make sense to employ them”. like, the former has probably happened multiple times in history, whereas the latter is unprecedented. what you’re saying makes more sense to me if you have the former in mind, but we’re talking about the latter (“people being permanently unemployable”)
Yes, I am talking about 50% people being permanently unemployable, i.e., not being capable of doing any labor that someone would pay meaningful amounts for.
It seems to me that the crux between us is something like: I find a very jagged capability, “Moravec-ian” world plausible, i.e., AI can do lots/most of economically valuable stuff competently, with the amount of human oversight small enough to make 50% of humanity permanently unemployable, while still not being “human-level” on all axes and this remaining stable for a few years at least (which also touches on your claim 2, i.e., an AI that could do all this doesn’t yet exist).
But maybe I’m wrong, and you actually need to be way closer to “human-complete” to do all the boring economic tasks, and AI that is not near-human-complete would not be massively deployable to do them with minimal oversight.
I am now more uncertain, so I will somewhat revise my top-level comment.
I realized I’m not sure how you define “50% of people permanently unemployable”. Surely it isn’t about global population? Is it about global labor force (which is ~45% of global population) or about developed countries only?
As of 2019, about a quarter of global labor force worked in primary agricultural production (mostly smallholder farmers who might only be impacted by AI indirectly, such as natural gas going to data centers instead of fertilizer plants) and half as much were employed in “off-farm segments of agrifood systems”. Surely people need to eat and those jobs are here to stay.
Please define specifically, 50% of which people in particular, and what does “permanently unemployable” mean exactly (for example, what about a laid-off white-collar worker who can return to the parents’ village and get a job at a local shop or school?)
We have ways to measure unemployment. It classifies some people as unemployed and classifies some people as labor force, and the fraction of the former divided by the latter is the unemployment rate, which today hovers around 5% (depending on the country). 50% of people are permanently unemployable if they are too useless to leave that category by the time they stop counting as participating in the labor force.
(Also, to be clear, Toby was the first one who used the phrase “50% of people permanently unemployable” without defining clearly what he meant by this, and I responded to it without reflecting that maybe it’s good to clarify (and then Kaarel did the same after me). So, I appreciate you pushing me for clarification, but depending on your interest, you might care more about what Toby means by that, not what I mean by that.)
The most common reason people stop counting as participating in the labor force is that they grow old and living off savings, passive income, pension and/or social benefits is better than continuing working, which we call a retirement. With global graying of the population, 50% of formerly working people will necessarily become permanently unemployable in this sense eventually even without the AI progress.
Also, note that Finland has ~10% unemployment rate and they are quite OK because of the social safety net. If AI was to be heavily taxed and these funds were used to support the population suffering job losses (implausible for the US indeed but plausible for Europe), even in absence of “strong AGI” people might choose not to work in order to get welfare without actually being unemployable.
(Yeah I do care but Toby has not left a single comment here)
So I actually like the cloud metaphor. But as you say
it’s debatable whether we’re in the cloud, which means that perhaps we should be forecasting other things than “AGI”.
And you do give some more specific milestones, like
but (1) “the point where AI could take over from humanity were it misaligned”, (2) “[the point where] it has made 50% of people permanently unemployable”, and (3) “[the point where it] has doubled the global rate of technological progress” seem to me plausibly quite distinct. I
expect[ETA[1]: find it plausible for] (1) to come years after (2) and (3); and (3) probably before (2), but I have substantial uncertainty here. Maybe you could get (2) before (3) if you really crank boring LLM training the right way + robotics. And maybe powerful optimization is out there to screw us, and (1) comes before (2) and (3). If I were to forecast it in a principled probabilistic way, I would end up with three distinct, but somewhat dependent probability distributions.Worth marking that it’s a log-x plot.
see the thread below for details
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster, [1] a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI [2]
Conclusion. if you somehow don’t have wildly superhuman AI at the beginning of this thought experiment, you will have it in at most
(On the surface, it looks like I’m saying “if P then Q”, where P is (2) AND (3) AND not-(1) (using your numbering of events), and Q is (1)-soon, so making a claim about the conditional in which P is true. But that’s not really what I mean to say. Like, to me P is true in some sorts of weird magic worlds carrying (let’s say)
Another pathway that I have substantial mass on: [the first learning setup in which novel mental structure creation / novel domain learning starts to work basically at all] produces a wildly superhuman AI. Like, stuff gets crazy inside a single training/growth process.
absent strong AI regulation
actually, i’d guess this is true even with “at most 1 year” instead of “at most 10 years”
In such a case I expect these AI researchers to pick all the low- and medium-hanging fruit at the then-current compute level/hardware technology, and then the algorithmic progress gets saturated until new-gen chips are produced in quality. Check this: https://www.lesswrong.com/posts/sGNFtWbXiLJg2hLzK
Since my Claim 1 is about the conceptual work input being 100x sped up, not some final output being 100x sped up, I’ll take you to be disagreeing with Claim 2. So the question is: is 10 years of thinking about AI algorithms followed by 1 month of retraining sufficient [to get from AI that causes
I think we should imagine the fruits at the beginning of this to not have been well-picked (supposing a crazy smart AI does not already exist).
Trusting Byrnes’s decomposition of the 7 year 600x nanochat cost improvement, that’s 6x from hardware and 100x from non-hardware. That would give some sort of baseline guess of
Byrnes explicitly does not include algo ideas that “are not about doing the same thing more efficiently, but rather about trying to do something different instead”. See Section 1.5 of his post. But these clearly should be included in our context here, and are majorly important imo. E.g., curating curricula, creating problems for oneself in a different way, coming up with good ways to reward problem-creation, creating more nested levels of problem-solving with their own rewards, coming up with other ways to make rewards denser / track progress better, creating tools for oneself, various IDA ideas (beyond those already mentioned), etc.. There are also various ways humans get smarter over centuries and over a lifetime that should also count for our purposes as “algo progress” if the AIs can carry them out, e.g. inventing+acquiring new concepts, questions, methods, and skills, and just knowing more.
In our scenario, coming up with an arbitrarily different new AI design is also legitimate, as long as this AI can be created/trained/grown in at most 1 month.
Tbh a lot of my belief that you get a lot of progress just comes from it being an extremely high-dimensional design space and there surely being lots of things one can do so much better in there.
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
Unless humans are strictly needed to orchestrate the AIs, but a world where they have thinking coherent enough to make massive algorithmic progress, but incoherent enough to pursue this competently, seems super weird, but, hey, maybe Moravec will come to bite us again!
You mean something like serial, uninterrupted, focused thinking, like a WBE of a very-high-g AI researcher that doesn’t need sleep?
this makes me want to ask: are you tracking the difference between the event “50
(to make sure we’re on the same page: in my view, this is unlikely to be a somewhat stable situation)
Yes, I am talking about 50% people being permanently unemployable, i.e., not being capable of doing any labor that someone would pay meaningful amounts for.
It seems to me that the crux between us is something like: I find a very jagged capability, “Moravec-ian” world plausible, i.e., AI can do lots/most of economically valuable stuff competently, with the amount of human oversight small enough to make 50% of humanity permanently unemployable, while still not being “human-level” on all axes and this remaining stable for a few years at least (which also touches on your claim 2, i.e., an AI that could do all this doesn’t yet exist).
But maybe I’m wrong, and you actually need to be way closer to “human-complete” to do all the boring economic tasks, and AI that is not near-human-complete would not be massively deployable to do them with minimal oversight.
I am now more uncertain, so I will somewhat revise my top-level comment.
i just meant the usual human AI algo research community doing this for 10 years
I realized I’m not sure how you define “50% of people permanently unemployable”. Surely it isn’t about global population? Is it about global labor force (which is ~45% of global population) or about developed countries only?
As of 2019, about a quarter of global labor force worked in primary agricultural production (mostly smallholder farmers who might only be impacted by AI indirectly, such as natural gas going to data centers instead of fertilizer plants) and half as much were employed in “off-farm segments of agrifood systems”. Surely people need to eat and those jobs are here to stay.
Please define specifically, 50% of which people in particular, and what does “permanently unemployable” mean exactly (for example, what about a laid-off white-collar worker who can return to the parents’ village and get a job at a local shop or school?)
We have ways to measure unemployment. It classifies some people as unemployed and classifies some people as labor force, and the fraction of the former divided by the latter is the unemployment rate, which today hovers around 5% (depending on the country). 50% of people are permanently unemployable if they are too useless to leave that category by the time they stop counting as participating in the labor force.
(Also, to be clear, Toby was the first one who used the phrase “50% of people permanently unemployable” without defining clearly what he meant by this, and I responded to it without reflecting that maybe it’s good to clarify (and then Kaarel did the same after me). So, I appreciate you pushing me for clarification, but depending on your interest, you might care more about what Toby means by that, not what I mean by that.)
The most common reason people stop counting as participating in the labor force is that they grow old and living off savings, passive income, pension and/or social benefits is better than continuing working, which we call a retirement. With global graying of the population, 50% of formerly working people will necessarily become permanently unemployable in this sense eventually even without the AI progress.
Also, note that Finland has ~10% unemployment rate and they are quite OK because of the social safety net. If AI was to be heavily taxed and these funds were used to support the population suffering job losses (implausible for the US indeed but plausible for Europe), even in absence of “strong AGI” people might choose not to work in order to get welfare without actually being unemployable.
(Yeah I do care but Toby has not left a single comment here)