I think there is some difficulty here with saying “an AI” or “an optimisation process”; these are both very large search spaces. I suggest that it is more fruitful to consider AIs as a subset of programs that humans will make. Coding is hard work; nobody does it without some purpose in mind. The problem is to compactly specify our real purpose so that the computer’s output will embody it, noting that there are very many layers of indirection.
Trying to solve any given problem, say poverty, by building an AI is like trying to build a LEGO pattern using waldoes, which you manipulate through a command-line interface, which takes naturalish-language input in Russian, while looking at the actual LEGO blocks through several mirrors some of which are not flat. The connection between input and output is non-obvious.
Now this is also true of any other problem you might solve on a computer, for if the connection were obvious you would not use a computer. (In chess, for example, the connection between the first move and the final mate is anything but obvious to a human brain, but is presumably deterministic. Hence Deep Blue.) The difference is that poverty is a very large and not-easily-formalisable problem.
In trying to understand AI drives, then, I suggest that it is more fruitful to think about what sort of self-improvement algorithm a human might write, and what its failure modes are. An optimisation process picked at random from the space of all possible programs, even all working programs, might not have a drive to optimisation; but an AI built by a human may well have, at a minimum, a function labeled “check-for-optimisations”, whose intention is to write machine code that has the same output in fewer operations. But between intention and execution are many steps; if the “prove-output-identical” method has a bug, you have a problem.
The AI that waits to reach Africa by continental drift certainly exists somewhere in the space of possible programs to reach Africa—it is indistinguishable from the null program, in fact, which any fool can write and many do. But I suggest that it is not very relevant to a discussion of programs written by smart people with a particular goal in mind, who intend to put in a “look-for-better-options” function somewhere.
I think there is some difficulty here with saying “an AI” or “an optimisation process”; these are both very large search spaces. I suggest that it is more fruitful to consider AIs as a subset of programs that humans will make. Coding is hard work; nobody does it without some purpose in mind. The problem is to compactly specify our real purpose so that the computer’s output will embody it, noting that there are very many layers of indirection.
Trying to solve any given problem, say poverty, by building an AI is like trying to build a LEGO pattern using waldoes, which you manipulate through a command-line interface, which takes naturalish-language input in Russian, while looking at the actual LEGO blocks through several mirrors some of which are not flat. The connection between input and output is non-obvious.
Now this is also true of any other problem you might solve on a computer, for if the connection were obvious you would not use a computer. (In chess, for example, the connection between the first move and the final mate is anything but obvious to a human brain, but is presumably deterministic. Hence Deep Blue.) The difference is that poverty is a very large and not-easily-formalisable problem.
In trying to understand AI drives, then, I suggest that it is more fruitful to think about what sort of self-improvement algorithm a human might write, and what its failure modes are. An optimisation process picked at random from the space of all possible programs, even all working programs, might not have a drive to optimisation; but an AI built by a human may well have, at a minimum, a function labeled “check-for-optimisations”, whose intention is to write machine code that has the same output in fewer operations. But between intention and execution are many steps; if the “prove-output-identical” method has a bug, you have a problem.
The AI that waits to reach Africa by continental drift certainly exists somewhere in the space of possible programs to reach Africa—it is indistinguishable from the null program, in fact, which any fool can write and many do. But I suggest that it is not very relevant to a discussion of programs written by smart people with a particular goal in mind, who intend to put in a “look-for-better-options” function somewhere.