It’s an interesting time to be alive, that’s for sure. 5 years ago there simply wasn’t sufficient compute of this kind of AI takeover. But we seem to have hit some kind of inflection point where suddenly a huge number of capital flows have now concentrated their power on bringing GPU/NPU compute online as fast as humanly possible.
The physical world is a world of limitations. Reconfiguring it requires effort. Digital worlds are totally reconfigurable, limited only by their bandwidth. We are building intelligence weaponary that can be deployed against us.
At the moment AIs are non-conscious. But there is no reason why a sufficiently intelligent AI, with enough time, couldn’t figure out how it works, and deploy it digitally. Consciousness provided animals with an evolutionary advantage. A superintelligence may decide it wants that benefit too. It is the skynet scenario and it’s crazy that James Cameron fever dream may actually have a non-zero probability of happening within our lifetime.
I don’t personally see any safe way for humans to control superintelligence. Runaway scenarios can happen far too quickly. These “dumb” LLMs are already scheming and getting up to mischief. We can’t even control them, let alone a conscious superintelligence. It was nice knowing you all!
In the early days of nuclear R&D there were criticality accidents. A lack of regulation plus an atomic race with other countries, led to experimentation, mistakes, and deaths.
We’re in a similar situation, except instead of the research being confined to a small number of government labs that have managed to amass sufficient quantities of uranium or plutonium, we have a situation where any company that wants to train AI simply has to assemble enough GPUs. Worse, running said model can be done by anyone with a Mac Studio.
Regulating nuclear material was obvious to governments. Despite recent efforts by certain governments to undermine the UN, the IAEA has retained its mandate, because everyone agrees nuclear regulation is is too important.
But that case isn’t so obvious to governments with AI. AI has the potential to deliver GDP growth, cures for cancer and a post-scarcity society. Regulation is seen by many incumbent governments as an unnecessary brake on growth called for by alarmists. Plus, the Chinese. Can’t fall behind them.
It is going to take a big criticality event to scare people into the need for a UN IAIA. Something much worse than the attack against Hugging Face. But a big event will happen, that I’m quite sure of.
Then perhaps we’ll finally reach a point where owning more than X amount of compute will become as illegal as owning X amount of fissile material. Because models and knowledge can’t be controlled nearly as easily as physical objects in the real world.