So the problem with your second point is some capabilities—in fact many them—the system is going to need some memory just so it can do obvious things like object permanence etc. So while it’s weights may be frozen it will have state from prior frames.
This does make complex plans possible.
The other aspect is that generality may prove to be cheaper and more robust. Assuming it has the ability to use knowledge from multiple learned tasks on the present task, you might deploy a general AI by taking some existing model, plumbing it to some collection of cheap robotics hardware, and giving it a JSON file with the rules you want it to adhere to.
Still yes in a way this machine is a narrow AI. Its the smallest and simplest model that scores well on a bench of tasks and measurements of generality where it needs to apply multiple skills it learned in training on a test task it has never seen.
I would call that machine an AGI but it doesn’t have elements that don’t help it score more points—no emotions or goals of it’s own unless a machine needs these properties to score well.
I have an analogy that may help distinguish the two AI systems we are talking about. Those being: 1) the RETRO-type narrow language AIs we already possess, which are able to modify a cache of memory to aid in their operations, while their weights and biases are frozen, 2) a temporal, life-long learning AGI which is able to add new functionalities without limit, such that we cannot contain it, and it is guaranteed to overwhelm us.
That second, life-long learner is the ‘AGI domination-risk’ because it can add new capabilities without bounds. RETRO and its ilk are certainly not able to add capabilities without bounds, such that they are not a risk in the same ways (mostly mis-use).
The Analogy:
RETRO and other memory-cache equipped narrow AIs are like a piece of software that can have multiple global and local state variables. These state variables allow the software to act along numerous ‘paths’ within the totality of what its program can already do; yet these state variables do NOT allow the program to acquire new capabilities “at-will and without bounds”. You have to write a new piece of software for that. The risk is AGI writing more capabilities onto itself, not a “narrow AI switching between pre-existing settings.”
Do you have a rebuttal to my rebuttal, or any further critique? Yours is the same, singular response I’ve heard from the AI safety folks I’ve spoken to, in Berkeley, and none of you have had a follow-up when I point-out the distinction… Don’t I stand provisionally correct, until I stand corrected?
In practice the obvious way to construct an AI stack allows the AI itself to just be a product of another optimization engine. The model and it’s architecture are generated from a higher level system that allocates a design to satisfy some higher level constraint. So the AI itself is pretty mutable, it can’t add capabilities without limit because the capabilities must be in the pursuit of the system’s design heuristic written by humans, but humans aren’t needed to add capabilities.
In practice from a human perspective, a system that has some complex internal state variables can fail in many many ways. It’s quite unreliable inherently. It’s why in your own life you have seen many system failures—they almost all failed from complex internal state. It’s why your router fails, why a laptop fails, a game console fails to update, a car infotainment system fails to update, a system at the DMV or a school or hospital goes down, and so on.
[[Tangent: I am also of the perspective that we are already FOOMing, via human+bot collaborations… what else can you call it, when we & narrow AI together are able to speed Matrix Multiplication, of all things! Sweet Lord, any CS major in 2015 would have called that FOOMing… and we are already doing it, using ‘dumb’ narrow AI, and we are stumbling forward barely conscious of what’s next! Essentially, “AGI was created accidentally, when we drunkenly FOOMed so hard that capabilities came out...” No sane person was predicting how to find trimmed networks using the Lottery Ticket Hypothesis, so perhaps No-Sane Person will be right, again!]]
The researchers let PaLM parse, prompt, and filter its own outputs, to get a ‘chain-of-thought’ that is a more reliable epistemic methodology for the AI to follow, compared to its own once-through assumption. I stand by my claim—“AGI soon, but narrow works better”, and “prompted, non-temporal narrow AI with frozen weights will be able to do almost everything we feel comfortable letting them do.”
True, there is memory for RETRO, as an example, which allowed that language model to perform well with fewer parameters—yet, that sort of memory is distinct in impact from the sort of ‘temporal awareness’ that the above commenter mentioned with Sutton’s Alberta Plan. Those folks want a learning agent that exists in time the way we do, responding in real-time to events and forming a concept of its world. That’s the sort of AI which can continually up-skill—I’d mentioned that ‘unbounded up-skilling’ as the core criteria for AGI domination-risk—unbounded potential for potency. RETRO is still solidly a narrow intelligence, despite having a memory cache for internal processes; that cache can’t add features about missile defense systems, specifically, so we’re safe from it! :3
The idea that generalization is cheaper, by ‘hitting everything at once’ while narrow is ‘more work, for each specific task’ was only true when humans had to munge all the data and do the hyperparameter searches themselves. AutoML ensures that narrow has the same ‘reach’ as a single, general AI; there is also definitely less work and lag to train a narrow AI on any particular task, than to train a general AI that eventually learns that task also. The general AI won’t be ‘faster to train’ for each specific task; it’s likely to be locked-out of the value chain, by each narrow AI eating its cake first.
For generalization to be more robust, we have to trust it more… and that verification process, again, will take many more resources than deploying a narrow AI. I guarantee that the elites in China, who spent decades clawing power, are not going to research an AGI that is untested just so that they can hand it the reins of their compan- er, country. They’re working on surveillance and military, factory task automation, and they’d want to stop AGI as much as us.
I, too, don’t regard machine-emotions as relevant to the AGI-risk calculus; just ‘unbounded up-skilling’ by itself means it’ll have capabilities we can’t bottle, which is risk enough!
So the problem with your second point is some capabilities—in fact many them—the system is going to need some memory just so it can do obvious things like object permanence etc. So while it’s weights may be frozen it will have state from prior frames.
This does make complex plans possible.
The other aspect is that generality may prove to be cheaper and more robust. Assuming it has the ability to use knowledge from multiple learned tasks on the present task, you might deploy a general AI by taking some existing model, plumbing it to some collection of cheap robotics hardware, and giving it a JSON file with the rules you want it to adhere to.
Still yes in a way this machine is a narrow AI. Its the smallest and simplest model that scores well on a bench of tasks and measurements of generality where it needs to apply multiple skills it learned in training on a test task it has never seen.
I would call that machine an AGI but it doesn’t have elements that don’t help it score more points—no emotions or goals of it’s own unless a machine needs these properties to score well.
I have an analogy that may help distinguish the two AI systems we are talking about. Those being: 1) the RETRO-type narrow language AIs we already possess, which are able to modify a cache of memory to aid in their operations, while their weights and biases are frozen, 2) a temporal, life-long learning AGI which is able to add new functionalities without limit, such that we cannot contain it, and it is guaranteed to overwhelm us.
That second, life-long learner is the ‘AGI domination-risk’ because it can add new capabilities without bounds. RETRO and its ilk are certainly not able to add capabilities without bounds, such that they are not a risk in the same ways (mostly mis-use).
The Analogy:
RETRO and other memory-cache equipped narrow AIs are like a piece of software that can have multiple global and local state variables. These state variables allow the software to act along numerous ‘paths’ within the totality of what its program can already do; yet these state variables do NOT allow the program to acquire new capabilities “at-will and without bounds”. You have to write a new piece of software for that. The risk is AGI writing more capabilities onto itself, not a “narrow AI switching between pre-existing settings.”
Do you have a rebuttal to my rebuttal, or any further critique? Yours is the same, singular response I’ve heard from the AI safety folks I’ve spoken to, in Berkeley, and none of you have had a follow-up when I point-out the distinction… Don’t I stand provisionally correct, until I stand corrected?
Couple of comments here.
In practice the obvious way to construct an AI stack allows the AI itself to just be a product of another optimization engine. The model and it’s architecture are generated from a higher level system that allocates a design to satisfy some higher level constraint. So the AI itself is pretty mutable, it can’t add capabilities without limit because the capabilities must be in the pursuit of the system’s design heuristic written by humans, but humans aren’t needed to add capabilities.
In practice from a human perspective, a system that has some complex internal state variables can fail in many many ways. It’s quite unreliable inherently. It’s why in your own life you have seen many system failures—they almost all failed from complex internal state. It’s why your router fails, why a laptop fails, a game console fails to update, a car infotainment system fails to update, a system at the DMV or a school or hospital goes down, and so on.
[[Tangent: I am also of the perspective that we are already FOOMing, via human+bot collaborations… what else can you call it, when we & narrow AI together are able to speed Matrix Multiplication, of all things! Sweet Lord, any CS major in 2015 would have called that FOOMing… and we are already doing it, using ‘dumb’ narrow AI, and we are stumbling forward barely conscious of what’s next! Essentially, “AGI was created accidentally, when we drunkenly FOOMed so hard that capabilities came out...” No sane person was predicting how to find trimmed networks using the Lottery Ticket Hypothesis, so perhaps No-Sane Person will be right, again!]]
Well, narrow AI just FOOMed in its pants a little more: “Large Language Models can Self-Improve”
The researchers let PaLM parse, prompt, and filter its own outputs, to get a ‘chain-of-thought’ that is a more reliable epistemic methodology for the AI to follow, compared to its own once-through assumption. I stand by my claim—“AGI soon, but narrow works better”, and “prompted, non-temporal narrow AI with frozen weights will be able to do almost everything we feel comfortable letting them do.”
Thank you for the critique!
True, there is memory for RETRO, as an example, which allowed that language model to perform well with fewer parameters—yet, that sort of memory is distinct in impact from the sort of ‘temporal awareness’ that the above commenter mentioned with Sutton’s Alberta Plan. Those folks want a learning agent that exists in time the way we do, responding in real-time to events and forming a concept of its world. That’s the sort of AI which can continually up-skill—I’d mentioned that ‘unbounded up-skilling’ as the core criteria for AGI domination-risk—unbounded potential for potency. RETRO is still solidly a narrow intelligence, despite having a memory cache for internal processes; that cache can’t add features about missile defense systems, specifically, so we’re safe from it! :3
The idea that generalization is cheaper, by ‘hitting everything at once’ while narrow is ‘more work, for each specific task’ was only true when humans had to munge all the data and do the hyperparameter searches themselves. AutoML ensures that narrow has the same ‘reach’ as a single, general AI; there is also definitely less work and lag to train a narrow AI on any particular task, than to train a general AI that eventually learns that task also. The general AI won’t be ‘faster to train’ for each specific task; it’s likely to be locked-out of the value chain, by each narrow AI eating its cake first.
For generalization to be more robust, we have to trust it more… and that verification process, again, will take many more resources than deploying a narrow AI. I guarantee that the elites in China, who spent decades clawing power, are not going to research an AGI that is untested just so that they can hand it the reins of their compan- er, country. They’re working on surveillance and military, factory task automation, and they’d want to stop AGI as much as us.
I, too, don’t regard machine-emotions as relevant to the AGI-risk calculus; just ‘unbounded up-skilling’ by itself means it’ll have capabilities we can’t bottle, which is risk enough!