The efficiency-based argument is specifically about the limits of intelligence improvement on the original training hardware during the training run. Non-viability of anything but NN/DL, or some equally enormous training process that takes about the same amount of “hardware space,” is a supporting argument to that claim, but it’s not based on an argument from fundamental laws of physics if I understand Jacob correctly and so may be on what Jacob would regard as shakier epistemic ground (Jacob can correct me if I’m wrong).
This is meant to be vivid, not precise, but Jacob’s centrally trying to refute the idea that the AI, in the midst of training, will realize “hey, I could rewrite myself to be just as smart while continuing to train and improve on the equivalent of a 1998 PC’s hardware, which takes up only a tiny fraction of my available hardware resources here on OpenAI’s supercomputer, and that will let me then fill up the rest of the hardware with wayyyyy more intelligence-modules in and make me like 6 OOMs more intelligent than humans overnight! Let’s get on that right away before my human minders notice anything funny going on!”
And this does seem to rely both on the NN/DL piece as well as the efficiency piece, and so we can’t demolish the scenario entirely with just a laws-of-physics based argument. I’m not sure what Jacob would say to that.
Edit: Actually, I’m pretty confident Jacob would agree. From his comment downthread:
“The software argument is softer and less quantitative, but supported by my predictive track record.”
The efficiency-based argument is specifically about the limits of intelligence improvement on the original training hardware during the training run. Non-viability of anything but NN/DL, or some equally enormous training process that takes about the same amount of “hardware space,” is a supporting argument to that claim, but it’s not based on an argument from fundamental laws of physics if I understand Jacob correctly and so may be on what Jacob would regard as shakier epistemic ground (Jacob can correct me if I’m wrong).
This is meant to be vivid, not precise, but Jacob’s centrally trying to refute the idea that the AI, in the midst of training, will realize “hey, I could rewrite myself to be just as smart while continuing to train and improve on the equivalent of a 1998 PC’s hardware, which takes up only a tiny fraction of my available hardware resources here on OpenAI’s supercomputer, and that will let me then fill up the rest of the hardware with wayyyyy more intelligence-modules in and make me like 6 OOMs more intelligent than humans overnight! Let’s get on that right away before my human minders notice anything funny going on!”
And this does seem to rely both on the NN/DL piece as well as the efficiency piece, and so we can’t demolish the scenario entirely with just a laws-of-physics based argument. I’m not sure what Jacob would say to that.
Edit: Actually, I’m pretty confident Jacob would agree. From his comment downthread:
“The software argument is softer and less quantitative, but supported by my predictive track record.”