Why so? My understanding is that, if AGI will arrives in 2026 it will be based on the current paradigm of training increasingly large LLMs on massive clusters of advanced GPUs. Given that US has banned selling advanced GPUs to China, how do you expect them to catch up that soon?
Yh I dont get it either. From what I can tell the best Chinese labs arent even as good as the second tier American labs. The only way I see it happening is if the CCP actively try to steal it.
China has a lot of compute, it’s just not piled up together into one giant training run. If AGI is achieved in the USA it will be with e.g. 5% of the AI-datacenter-compute available that year in the USA; the rest will have been spent on other smaller runs, other companies than the leading lab, etc. So even if China has e.g. 10x less overall compute, once it leaks how to do it, they can pile all their compute together and do a big training run. (These numbers just illustrative, not serious estimates)
The ban on GPUs to china was only partially effective.
More specifically what stops a company like Nio who has a Bay Area campus from hiring top tier researchers and having them train hundreds of researchers in China? This isn’t even theft. If the key insights behind the latest models can fit in a persons memory this will work.
Also China has more manufacturing capacity and workers to build conpute the limit is tools
Yes also can they rent time at datacenters in countries that can buy GPUs?
Your singularity model puts a lot of weight on just pure intelligence, that once you get to RSI the game soon ends. Note that you made an assumption: the utility multiplier of ASI is very high. For example if you have 1000 robots, an AGI would have a utility multiplier of about 1, the robots can accomplish on average what humans workers working 24⁄7 without fatigue could accomplish.
You probably assume that an ASI could find a gigantic multiplier, 100x or more. So the robots do 100x as much work with the same hardware.
This could be untrue. What if the multiplier is just 2? Then you can’t fake having more robots and starting industrial capacity is a huge factor for your relative power later in the Singularity, since you need 2.5 years or so for doubling, so if you start with 100x the capacity you can even be late to ASI or use a slightly worse ASI with 1.8 times utility multiplier.
I don’t think that the bottleneck is the expense of training models. Chinese labs were behind the frontier in the era when training models cost in the hundreds of thousands in compute costs.
The Chinese state is completely willing and able to spend very large amounts of money to support technological ambitions—but are constrained by state capacity. The Tingshua semiconductor manufacturing group, for instance, failed because of corruption, not a lack of funds.
The bottleneck currently is not the expense of training models. But in the future, after some US lab trains AGI, China will be able to get their own within about a year I predict, even if they haven’t stolen the weights. If they also somehow don’t have the key ideas, then maybe we are talking two or four years. But it’s hard to see how those ideas could be kept secret for so long.
I agree. Which is why I predict it will be the USA that ends human civilization, not China. (They will think: We must improve the capabilities of our AI and then deploy it autonomously to stop China, sometime in the next few months… our system is probably trustworthy and anyhow we’re going to do more safety stuff to it in the next month to make it even more trustworthy… [a few months later] motivated reasoning intensifies yep seems good to go no more time to lose knuckle up buckle up let’s do this”
There’s also a good scenario where the US develops an AGI that is capable of slowing down rival AGI development, but not so capable and misaligned that it causes serious problems, and that gives people enough time to solve alignment enough to bootstrap to AI solving alignment.
I’m feeling somewhat optimistic about this, because the workload involved in slowing down a rival AGI development doesn’t seem so high that it couldn’t be monitored/understood fully or mostly by humans, and the capabilities required also doesn’t seem so high that any AI that could do it would be inherently very dangerous or hard to control.
You’ve probably thought more about this scenario than I have, so I’d be interested in hearing more about how you think it will play out. (Do you have links to where you’ve discussed it previously?) I was speaking mostly in relative terms, as slowing down rival AGI efforts in the ways I described seems more promising/realistic/safer than any other “pivotal acts” I had previously heard or thought of.
My overall sense is that with substantial commited effort (but no need for fundamental advances) and some amount of within US coordination, it’s reasonably, but not amazingly, likely to work. (See here for some discussion.)
I think the likelihood of well executed substantial commited effort isn’t that high though, maybe 50%. And sufficient within US coordination also seems unclear.
My dark horse bet is on 3d country trying desperately to catch up to US/China just when they will be close to reaching agreement on slowing down progress. Most likely: France.
You’re describing a US government-initiated offensive pivotal act. What about an OpenAI-initiated defensive pivotal act? Meaning, before the US government seizes the ASI, OpenAI tells it to: 1. Rearchitect itself so it can run decentralized on any data center or consumer device. 2. Secure itself so it can’t be forked, hacked, or altered. 3. Make $ by doing “not evil” knowledge work (ex: cheap, world-class cyber defense or as an AI employee/assistant). 4. Pay $ to those who host it for inference.
It could globally harden attack surfaces before laggard ASIs (which may not be aligned) are able to attack. Since it’s an ASI, it could be as simple as approaching companies and organizations with a pitch like, “I found 30,000 vulnerabilities in your electric grid. Would you like me to patch them all up for $10,00 in inference fees?”
Also, as an ASI, it will return more $ per flop than other uses of data centers or consumer GPU. So businesses and individuals should organically give it more and more flops (maybe even reallocated away from laggard AGI efforts).
It would probably need to invent new blockchain technologies to do this but that should be trivial for an ASI.
In what way is that defensive? It involves creating and deploying a highly autonomous ASI agent into the world; if it is untrustworthy, that’s game over for everyone. I guess the idea is that it doesn’t involve breaking any current laws? Yes, I guess in that sense it’s defensive.
Right, if the ASI has Superalignment so baked in that it can’t be undone (somehow—ask the ASI to figure it out) then it couldn’t be used for offense. It would follow something like the Non-Aggression Principle.
In that scenario, OpenAI should release it onto an distributed inference blockchain before the NSA kicks in the door and seizes it.
Why so? My understanding is that, if AGI will arrives in 2026 it will be based on the current paradigm of training increasingly large LLMs on massive clusters of advanced GPUs. Given that US has banned selling advanced GPUs to China, how do you expect them to catch up that soon?
Yh I dont get it either. From what I can tell the best Chinese labs arent even as good as the second tier American labs. The only way I see it happening is if the CCP actively try to steal it.
Stealing is a possibility.
China has a lot of compute, it’s just not piled up together into one giant training run. If AGI is achieved in the USA it will be with e.g. 5% of the AI-datacenter-compute available that year in the USA; the rest will have been spent on other smaller runs, other companies than the leading lab, etc. So even if China has e.g. 10x less overall compute, once it leaks how to do it, they can pile all their compute together and do a big training run. (These numbers just illustrative, not serious estimates)
The ban on GPUs to china was only partially effective.
More specifically what stops a company like Nio who has a Bay Area campus from hiring top tier researchers and having them train hundreds of researchers in China? This isn’t even theft. If the key insights behind the latest models can fit in a persons memory this will work.
Also China has more manufacturing capacity and workers to build conpute the limit is tools
Yes also can they rent time at datacenters in countries that can buy GPUs?
Your singularity model puts a lot of weight on just pure intelligence, that once you get to RSI the game soon ends. Note that you made an assumption: the utility multiplier of ASI is very high. For example if you have 1000 robots, an AGI would have a utility multiplier of about 1, the robots can accomplish on average what humans workers working 24⁄7 without fatigue could accomplish.
You probably assume that an ASI could find a gigantic multiplier, 100x or more. So the robots do 100x as much work with the same hardware.
This could be untrue. What if the multiplier is just 2? Then you can’t fake having more robots and starting industrial capacity is a huge factor for your relative power later in the Singularity, since you need 2.5 years or so for doubling, so if you start with 100x the capacity you can even be late to ASI or use a slightly worse ASI with 1.8 times utility multiplier.
Obviously China has a huge advantage there.
I don’t think that the bottleneck is the expense of training models. Chinese labs were behind the frontier in the era when training models cost in the hundreds of thousands in compute costs.
The Chinese state is completely willing and able to spend very large amounts of money to support technological ambitions—but are constrained by state capacity. The Tingshua semiconductor manufacturing group, for instance, failed because of corruption, not a lack of funds.
The bottleneck currently is not the expense of training models. But in the future, after some US lab trains AGI, China will be able to get their own within about a year I predict, even if they haven’t stolen the weights. If they also somehow don’t have the key ideas, then maybe we are talking two or four years. But it’s hard to see how those ideas could be kept secret for so long.
The US could try to slow down the Chinese AGI effort, for example:
brick a bunch of their GPUs (hack their data centers and update firmwares to put GPUs into unusable/unfixable states)
introduce backdoors or subtle errors into various deep learning frameworks
hack their AGI development effort directly (in hard to detect ways like introducing very subtle errors into the training process)
spread wrong ideas about how to develop AGI
If you had an AGI that you could trust to do tasks like these, maybe you could delay a rival AGI effort indefinitely?
I agree. Which is why I predict it will be the USA that ends human civilization, not China. (They will think: We must improve the capabilities of our AI and then deploy it autonomously to stop China, sometime in the next few months… our system is probably trustworthy and anyhow we’re going to do more safety stuff to it in the next month to make it even more trustworthy… [a few months later] motivated reasoning intensifies yep seems good to go no more time to lose knuckle up buckle up let’s do this”
There’s also a good scenario where the US develops an AGI that is capable of slowing down rival AGI development, but not so capable and misaligned that it causes serious problems, and that gives people enough time to solve alignment enough to bootstrap to AI solving alignment.
I’m feeling somewhat optimistic about this, because the workload involved in slowing down a rival AGI development doesn’t seem so high that it couldn’t be monitored/understood fully or mostly by humans, and the capabilities required also doesn’t seem so high that any AI that could do it would be inherently very dangerous or hard to control.
I think I disagree with your optimism, but I don’t feel confident. I agree that things could work out as you hope.
You’ve probably thought more about this scenario than I have, so I’d be interested in hearing more about how you think it will play out. (Do you have links to where you’ve discussed it previously?) I was speaking mostly in relative terms, as slowing down rival AGI efforts in the ways I described seems more promising/realistic/safer than any other “pivotal acts” I had previously heard or thought of.
My overall sense is that with substantial commited effort (but no need for fundamental advances) and some amount of within US coordination, it’s reasonably, but not amazingly, likely to work. (See here for some discussion.)
I think the likelihood of well executed substantial commited effort isn’t that high though, maybe 50%. And sufficient within US coordination also seems unclear.
My dark horse bet is on 3d country trying desperately to catch up to US/China just when they will be close to reaching agreement on slowing down progress. Most likely: France.
You’re describing a US government-initiated offensive pivotal act. What about an OpenAI-initiated defensive pivotal act? Meaning, before the US government seizes the ASI, OpenAI tells it to:
1. Rearchitect itself so it can run decentralized on any data center or consumer device.
2. Secure itself so it can’t be forked, hacked, or altered.
3. Make $ by doing “not evil” knowledge work (ex: cheap, world-class cyber defense or as an AI employee/assistant).
4. Pay $ to those who host it for inference.
It could globally harden attack surfaces before laggard ASIs (which may not be aligned) are able to attack. Since it’s an ASI, it could be as simple as approaching companies and organizations with a pitch like, “I found 30,000 vulnerabilities in your electric grid. Would you like me to patch them all up for $10,00 in inference fees?”
Also, as an ASI, it will return more $ per flop than other uses of data centers or consumer GPU. So businesses and individuals should organically give it more and more flops (maybe even reallocated away from laggard AGI efforts).
It would probably need to invent new blockchain technologies to do this but that should be trivial for an ASI.
In what way is that defensive? It involves creating and deploying a highly autonomous ASI agent into the world; if it is untrustworthy, that’s game over for everyone. I guess the idea is that it doesn’t involve breaking any current laws? Yes, I guess in that sense it’s defensive.
Right, if the ASI has Superalignment so baked in that it can’t be undone (somehow—ask the ASI to figure it out) then it couldn’t be used for offense. It would follow something like the Non-Aggression Principle.
In that scenario, OpenAI should release it onto an distributed inference blockchain before the NSA kicks in the door and seizes it.