I’m surprised nobody put this problem in terms of optimization and “steering the future” (including Eliezer, though I suppose he might have tried to make a different point in his post).
As I see it, robots are a special case of machines intended to steer things in their immediate vicinity towards some preferred future. (The special case is that their acting parts and steering parts are housed in the same object, which is not terribly important, except that the subsumption architecture implies it.)
“Smart” robots have a component analogue to a cortex, which gathers sensory info, models the world, makes plans and directs the acting parts to do things. (This is true even if this component is just 50 lines of code compiled by a smart compiler.) The subsumption-based robots just contain many parts that are hard-wired with “reflexes” for every little connection between them, in such a way that “it just happens” that the ensemble acts intelligently, i.e. steers the future towards some intended subspace.
The fallacy I think is that the “just happens” part is not true. Some process has to generate “optimization rules”; even the very simple reflexes between the components have to be correct (meaning that in the real world steer the future towards something). Subsumption architecture fans will look at something like ants, notice that each element has very simple rules but that the colony works extremely well, and will say “Hey, those are simple rules, I can understand them, I bet I could create another set of simple rules to do what I want. I don’t need to build a complex brain.”
The problem is that “set of simple rules” is not the same as “simple set of rules”: in the ants’ case, millions of years of evolution led to a very complicated set of very simple rules that work well. A subsumtion-oriented programmer can only build a simple set of simple rules; a complex one won’t work.
To give an example, take the “move leg higher” case:
*) In a few millions of years of evolution, creatures whose legs went higher up when hitting obstacles (say, because of random interactions between unrelated cells) got to eat more and be eaten less; so genetic programs for legged creature tend to contain rules like that.
*) A subsumtion-oriented programmer may notice that a leg that hits something should lift higher (either because (a) he thought about it or because (2) he noticed that’s what ants or caterpillars do).
*) A general AI researcher might think about a brain smart enough to decide it should lift a leg over obstacles it wants to cross.
Of course the first programmer will notice that his solution to the “going over obstacles” problem is simpler than building a brain, and it would seem more efficient.
But it’s not more efficient, because (in case a) his very complex, general Natural Intelligence brain thought about it, or (in case b) millions of years of evolution caused this particular adaptation (and his general NI noticed it was useful). There’s also the problem that there are thousands of other things even a simple organism does, and thus thousands (or more) reflexes to add. Either he’ll try to program them (case a). This is steering the future via a Rube Goldberg machine; it can work if you put enormous resources into it, but most likely it will blow in your face (or drop an anvil on you).
Or he’ll turn to evolving algorithms (simulating case b). That would probably work (it did until now), but he probably doesn’t have enough resources (natural evolution is a very inefficient optimizer). And even when he does, he won’t understand how it works.
Or he might notice that even nature found central-processing useful, several times (pretty much everything above worms has a brain of some kind, and brains of arthropods, cephalopods and vertebrates evolved separately). So he’ll turn back to centralized programming: hard as it is, it’s a more efficient investment than the alternative. Note that this doesn’t mean that you need fully-general AI to drive a car. You need all tools you can get your hands on. Saying “reflexes are enough” is like saying “I don’t need no tools, rocks are good enough”.
I’m surprised nobody put this problem in terms of optimization and “steering the future” (including Eliezer, though I suppose he might have tried to make a different point in his post).
As I see it, robots are a special case of machines intended to steer things in their immediate vicinity towards some preferred future. (The special case is that their acting parts and steering parts are housed in the same object, which is not terribly important, except that the subsumption architecture implies it.)
“Smart” robots have a component analogue to a cortex, which gathers sensory info, models the world, makes plans and directs the acting parts to do things. (This is true even if this component is just 50 lines of code compiled by a smart compiler.) The subsumption-based robots just contain many parts that are hard-wired with “reflexes” for every little connection between them, in such a way that “it just happens” that the ensemble acts intelligently, i.e. steers the future towards some intended subspace.
The fallacy I think is that the “just happens” part is not true. Some process has to generate “optimization rules”; even the very simple reflexes between the components have to be correct (meaning that in the real world steer the future towards something). Subsumption architecture fans will look at something like ants, notice that each element has very simple rules but that the colony works extremely well, and will say “Hey, those are simple rules, I can understand them, I bet I could create another set of simple rules to do what I want. I don’t need to build a complex brain.”
The problem is that “set of simple rules” is not the same as “simple set of rules”: in the ants’ case, millions of years of evolution led to a very complicated set of very simple rules that work well. A subsumtion-oriented programmer can only build a simple set of simple rules; a complex one won’t work.
To give an example, take the “move leg higher” case:
*) In a few millions of years of evolution, creatures whose legs went higher up when hitting obstacles (say, because of random interactions between unrelated cells) got to eat more and be eaten less; so genetic programs for legged creature tend to contain rules like that.
*) A subsumtion-oriented programmer may notice that a leg that hits something should lift higher (either because (a) he thought about it or because (2) he noticed that’s what ants or caterpillars do).
*) A general AI researcher might think about a brain smart enough to decide it should lift a leg over obstacles it wants to cross.
Of course the first programmer will notice that his solution to the “going over obstacles” problem is simpler than building a brain, and it would seem more efficient.
But it’s not more efficient, because (in case a) his very complex, general Natural Intelligence brain thought about it, or (in case b) millions of years of evolution caused this particular adaptation (and his general NI noticed it was useful). There’s also the problem that there are thousands of other things even a simple organism does, and thus thousands (or more) reflexes to add. Either he’ll try to program them (case a). This is steering the future via a Rube Goldberg machine; it can work if you put enormous resources into it, but most likely it will blow in your face (or drop an anvil on you).
Or he’ll turn to evolving algorithms (simulating case b). That would probably work (it did until now), but he probably doesn’t have enough resources (natural evolution is a very inefficient optimizer). And even when he does, he won’t understand how it works.
Or he might notice that even nature found central-processing useful, several times (pretty much everything above worms has a brain of some kind, and brains of arthropods, cephalopods and vertebrates evolved separately). So he’ll turn back to centralized programming: hard as it is, it’s a more efficient investment than the alternative. Note that this doesn’t mean that you need fully-general AI to drive a car. You need all tools you can get your hands on. Saying “reflexes are enough” is like saying “I don’t need no tools, rocks are good enough”.