(1) “the point where AI could take over from humanity were it misaligned”, (2) “[the point where] it has made 50% of people permanently unemployable”, and (3) “[the point where it] has doubled the global rate of technological progress” seem to me plausibly quite distinct. I expect (1) to come years after (2) and (3)
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster,[1] a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI[2]
Conclusion. if you somehow don’t have wildly superhuman AI at the beginning of this thought experiment, you will have it in at most .
(On the surface, it looks like I’m saying “if P then Q”, where P is (2) AND (3) AND not-(1) (using your numbering of events), and Q is (1)-soon, so making a claim about the conditional in which P is true. But that’s not really what I mean to say. Like, to me P is true in some sorts of weird magic worlds carrying (let’s say) probability in which I wouldn’t actually have a strong sense of whether Q. But I’m not really making a claim about the conditional, I’m more trying to argue that P AND not-Q doesn’t hang together (which could happen because P itself already doesn’t hang together, and indeed I think that). That is, it’s really an argument by contradiction that by default (1) doesn’t come much later than (2)+(3), not a claim about the conditional.)
Another pathway that I have substantial mass on: [the first learning setup in which novel mental structure creation / novel domain learning starts to work basically at all] produces a wildly superhuman AI. Like, stuff gets crazy inside a single training/growth process.
the pace of conceptual work on AI algorithms is like >100x faster
In such a case I expect these AI researchers to pick all the low- and medium-hanging fruit at the then-current compute level/hardware technology, and then the algorithmic progress gets saturated until new-gen chips are produced in quality. Check this: https://www.lesswrong.com/posts/sGNFtWbXiLJg2hLzK
Since my Claim 1 is about the conceptual work input being 100x sped up, not some final output being 100x sped up, I’ll take you to be disagreeing with Claim 2. So the question is: is 10 years of thinking about AI algorithms followed by 1 month of retraining sufficient [to get from AI that causes of people to be permanently unemployable to crazy smart AI]? In other words, if one is only going to be able to pick low-hanging and medium-hanging fruit in 10 years, is picking those sufficient to get to crazy smart AI from that point? I claim that the answer is yes; some quick points:
I think we should imagine the fruits at the beginning of this to not have been well-picked (supposing a crazy smart AI does not already exist).
Trusting Byrnes’s decomposition of the 7 year 600x nanochat cost improvement, that’s 6x from hardware and 100x from non-hardware. That would give some sort of baseline guess of for 10 years. Ok, but maybe we should apply some adjustments to the factors. In particular, what about data? On the one hand, it will be tough to collect a lot of data from humans quickly in our scenario. On the other hand, it will be very easy to collect [a lot more data [than we have from humans]] from AIs in this scenario, and by that point this will probably be overall better. On the first hand again, maybe we should imagine data not mattering so much at that point. On the second hand again, all things considered that’s actually conceptually correlated with fooming far past human level quickly. We should also apply some global adjustment down for having less time for experiments to run.
Byrnes explicitly does not include algo ideas that “are not about doing the same thing more efficiently, but rather about trying to do something different instead”. See Section 1.5 of his post. But these clearly should be included in our context here, and are majorly important imo. E.g., curating curricula, creating problems for oneself in a different way, coming up with good ways to reward problem-creation, creating more nested levels of problem-solving with their own rewards, coming up with other ways to make rewards denser / track progress better, creating tools for oneself, various IDA ideas (beyond those already mentioned), etc.. There are also various ways humans get smarter over centuries and over a lifetime that should also count for our purposes as “algo progress” if the AIs can carry them out, e.g. inventing+acquiring new concepts, questions, methods, and skills, and just knowing more.
In our scenario, coming up with an arbitrarily different new AI design is also legitimate, as long as this AI can be created/trained/grown in at most 1 month.
Tbh a lot of my belief that you get a lot of progress just comes from it being an extremely high-dimensional design space and there surely being lots of things one can do so much better in there.
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster, a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
Unless humans are strictly needed to orchestrate the AIs, but a world where they have thinking coherent enough to make massive algorithmic progress, but incoherent enough to pursue this competently, seems super weird, but, hey, maybe Moravec will come to bite us again!
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI
You mean something like serial, uninterrupted, focused thinking, like a WBE of a very-high-g AI researcher that doesn’t need sleep?
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
this makes me want to ask: are you tracking the difference between the event “50 of current human jobs are basically automated” and the event “50 of humans are such that it basically does not make sense to employ them”. like, the former has probably happened multiple times in history, whereas the latter is unprecedented. what you’re saying makes more sense to me if you have the former in mind, but we’re talking about the latter (“people being permanently unemployable”). i have significant probability that you are tracking this correctly already but wanted to check just in case
(I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.)
(to make sure we’re on the same page: in my view, this is unlikely to be a somewhat stable situation)
this makes me want to ask: are you tracking the difference between the event “50 of current human jobs are basically automated” and the event “50 of humans are such that it basically does not make sense to employ them”. like, the former has probably happened multiple times in history, whereas the latter is unprecedented. what you’re saying makes more sense to me if you have the former in mind, but we’re talking about the latter (“people being permanently unemployable”)
Yes, I am talking about 50% people being permanently unemployable, i.e., not being capable of doing any labor that someone would pay meaningful amounts for.
It seems to me that the crux between us is something like: I find a very jagged capability, “Moravec-ian” world plausible, i.e., AI can do lots/most of economically valuable stuff competently, with the amount of human oversight small enough to make 50% of humanity permanently unemployable, while still not being “human-level” on all axes and this remaining stable for a few years at least (which also touches on your claim 2, i.e., an AI that could do all this doesn’t yet exist).
But maybe I’m wrong, and you actually need to be way closer to “human-complete” to do all the boring economic tasks, and AI that is not near-human-complete would not be massively deployable to do them with minimal oversight.
I am now more uncertain, so I will somewhat revise my top-level comment.
Claim 1. if >50% of people are not employable anymore and technological progress is >2x faster, [1] a huge fleet of AIs will probably be doing a lot of AI research really well so the pace of conceptual work on AI algorithms is like >100x faster
Claim 2. at most 10 years of human algo thinking followed by a retraining run that takes at most a month would be sufficient to go from top-human-level AI to wildly superhuman AI [2]
Conclusion. if you somehow don’t have wildly superhuman AI at the beginning of this thought experiment, you will have it in at most
(On the surface, it looks like I’m saying “if P then Q”, where P is (2) AND (3) AND not-(1) (using your numbering of events), and Q is (1)-soon, so making a claim about the conditional in which P is true. But that’s not really what I mean to say. Like, to me P is true in some sorts of weird magic worlds carrying (let’s say)
Another pathway that I have substantial mass on: [the first learning setup in which novel mental structure creation / novel domain learning starts to work basically at all] produces a wildly superhuman AI. Like, stuff gets crazy inside a single training/growth process.
absent strong AI regulation
actually, i’d guess this is true even with “at most 1 year” instead of “at most 10 years”
In such a case I expect these AI researchers to pick all the low- and medium-hanging fruit at the then-current compute level/hardware technology, and then the algorithmic progress gets saturated until new-gen chips are produced in quality. Check this: https://www.lesswrong.com/posts/sGNFtWbXiLJg2hLzK
Since my Claim 1 is about the conceptual work input being 100x sped up, not some final output being 100x sped up, I’ll take you to be disagreeing with Claim 2. So the question is: is 10 years of thinking about AI algorithms followed by 1 month of retraining sufficient [to get from AI that causes
I think we should imagine the fruits at the beginning of this to not have been well-picked (supposing a crazy smart AI does not already exist).
Trusting Byrnes’s decomposition of the 7 year 600x nanochat cost improvement, that’s 6x from hardware and 100x from non-hardware. That would give some sort of baseline guess of
Byrnes explicitly does not include algo ideas that “are not about doing the same thing more efficiently, but rather about trying to do something different instead”. See Section 1.5 of his post. But these clearly should be included in our context here, and are majorly important imo. E.g., curating curricula, creating problems for oneself in a different way, coming up with good ways to reward problem-creation, creating more nested levels of problem-solving with their own rewards, coming up with other ways to make rewards denser / track progress better, creating tools for oneself, various IDA ideas (beyond those already mentioned), etc.. There are also various ways humans get smarter over centuries and over a lifetime that should also count for our purposes as “algo progress” if the AIs can carry them out, e.g. inventing+acquiring new concepts, questions, methods, and skills, and just knowing more.
In our scenario, coming up with an arbitrarily different new AI design is also legitimate, as long as this AI can be created/trained/grown in at most 1 month.
Tbh a lot of my belief that you get a lot of progress just comes from it being an extremely high-dimensional design space and there surely being lots of things one can do so much better in there.
This is very much unobvious to me, but now that you say this, I realize that I anchored too hard on a specific scenario where the world has gone very hard on just automating away all the economic tasks/roles that can be automated away with advanced robotics and LLMs+++, while humans largely coordinated this fleet in cases that they wouldn’t handle.
But generally, like, to grant the assumption, suppose that 60% are not employable and 40% are employable. Why is this 40% employable? (I think I also took this to be a somewhat stable situation, for some time, not a mean value theorem sort of thing.) Presumably, because there are things that AI still doesn’t do well. Maybe it’s “just” because robotics is annoyingly hard, but it sounds more plausible to me that (also) AI still is not human-thinking-complete, which makes me somewhat sceptical about this massive conceptual algorithm progress speedup.
Unless humans are strictly needed to orchestrate the AIs, but a world where they have thinking coherent enough to make massive algorithmic progress, but incoherent enough to pursue this competently, seems super weird, but, hey, maybe Moravec will come to bite us again!
You mean something like serial, uninterrupted, focused thinking, like a WBE of a very-high-g AI researcher that doesn’t need sleep?
this makes me want to ask: are you tracking the difference between the event “50
(to make sure we’re on the same page: in my view, this is unlikely to be a somewhat stable situation)
Yes, I am talking about 50% people being permanently unemployable, i.e., not being capable of doing any labor that someone would pay meaningful amounts for.
It seems to me that the crux between us is something like: I find a very jagged capability, “Moravec-ian” world plausible, i.e., AI can do lots/most of economically valuable stuff competently, with the amount of human oversight small enough to make 50% of humanity permanently unemployable, while still not being “human-level” on all axes and this remaining stable for a few years at least (which also touches on your claim 2, i.e., an AI that could do all this doesn’t yet exist).
But maybe I’m wrong, and you actually need to be way closer to “human-complete” to do all the boring economic tasks, and AI that is not near-human-complete would not be massively deployable to do them with minimal oversight.
I am now more uncertain, so I will somewhat revise my top-level comment.
i just meant the usual human AI algo research community doing this for 10 years