Because the AI’s goal doesn’t refer to a spot inside the computer running the AI. The AI just does formal math. You can think of the AI as a program that stops when it finds an integer N obeying a certain equation. Such a program won’t stop upon finding an integer N such that “returning N causes the creation of a glider that crashes into the computer and changes the representation of the equation so that N becomes a valid solution” or whatever. That N is not a valid solution to the original equation, so the program skips it and looks at the next one. Simple as that.
First, you defined the equation so that it included the computer and itself (that simulator it uses to think, and also self improve as needed).
Now you are changing the definitions so that the equation is something else. There’s a good post by Eliezer about being specific , which you are not. Go define the equation first.
Also, it is not a question about narrow AI. I can right now write an ‘AI’ that would try to find self replicating glider gun that tiles entire game of life with something. And yes, that AI may run inside the machine in game of life. The issue is, that’s more like ‘evil terrorists using protein folder simulator AI connected to automated genome lab to make plague’, than ‘the AI maximizes paperclips’.
You handwave too much, and the people who already accept premise, they like the handwave that sounds vaguely theoretic. Those who do not, aren’t too impressed, and are only annoyed.
You handwave too much, and the people who already accept premise
Or the people who understand the mathematics.
Cousin_it’s mathematics is correct, if counter-intuitive to those not used to thinking about quines. Whether it implies what he thinks it implies is a separate question as I discuss here.
Well, I assumed that he was building an AGI, and even agreed that it is entirely possible to rig the AI so that something the AI does inside a sim, gets replicated in the outside world. I even gave example: you make narrow AI that generates a virus mostly by simulated molecular interactions (and has some sim of the human immune system, people’s response to the world events, what WHO might do, and such) and wire it up to a virus making lab that can vent it’s produce into the air in the building or something edit: or best yet one that can mail samples to what ever addresses. That would be the AI that kills everyone. Including the AI itself in it’s sim would serve little functional role, and this AI won’t wirehead. It’s clear that the AGI risk is not about this.
edit: and to clarify, the problem with vague handwaving is that without defining what you handwave around, it is easy to produce stuff that is irrelevant, but appears relevant and math-y.
edit: hmm, seems that post with the virus making AI example didn’t get posted. Still, http://lesswrong.com/lw/bfj/evidence_for_the_orthogonality_thesis/68cf and http://lesswrong.com/lw/bfj/evidence_for_the_orthogonality_thesis/68eo convey the point. I’ve never said it is literally impossible to make a narrow AI that is rigged to tile the game world with blocks. It is, clearly, possible. One could make a glider gun iterator that finds the self replicating glider gun in the simulator, then some simple mechanisms set to make that gun in the real world. That is not a case of AI wanting to do something to the real world. That’s a glorified case of ‘my thermostat doesn’t wirehead’, to borrow from Will_Newsome.
Other issue is that one could immediately define some specific goal like ‘number of live cells’, and we could discuss this more specifically, instead of vague handwave about ill defined goal. But I can’t just define things narrowly for the other side of an argument. The wireheading is a problem of systems that can improve themselves. A system that can e.g. decide that it can’t figure out how to maximize live cells but it can prove some good theorems about four blocks.
Because the AI’s goal doesn’t refer to a spot inside the computer running the AI. The AI just does formal math. You can think of the AI as a program that stops when it finds an integer N obeying a certain equation. Such a program won’t stop upon finding an integer N such that “returning N causes the creation of a glider that crashes into the computer and changes the representation of the equation so that N becomes a valid solution” or whatever. That N is not a valid solution to the original equation, so the program skips it and looks at the next one. Simple as that.
First, you defined the equation so that it included the computer and itself (that simulator it uses to think, and also self improve as needed).
Now you are changing the definitions so that the equation is something else. There’s a good post by Eliezer about being specific , which you are not. Go define the equation first.
Also, it is not a question about narrow AI. I can right now write an ‘AI’ that would try to find self replicating glider gun that tiles entire game of life with something. And yes, that AI may run inside the machine in game of life. The issue is, that’s more like ‘evil terrorists using protein folder simulator AI connected to automated genome lab to make plague’, than ‘the AI maximizes paperclips’.
I’m bowing out of this discussion because it doesn’t seem to improve anyone’s understanding.
You handwave too much, and the people who already accept premise, they like the handwave that sounds vaguely theoretic. Those who do not, aren’t too impressed, and are only annoyed.
Or the people who understand the mathematics.
Cousin_it’s mathematics is correct, if counter-intuitive to those not used to thinking about quines. Whether it implies what he thinks it implies is a separate question as I discuss here.
Well, I assumed that he was building an AGI, and even agreed that it is entirely possible to rig the AI so that something the AI does inside a sim, gets replicated in the outside world. I even gave example: you make narrow AI that generates a virus mostly by simulated molecular interactions (and has some sim of the human immune system, people’s response to the world events, what WHO might do, and such) and wire it up to a virus making lab that can vent it’s produce into the air in the building or something edit: or best yet one that can mail samples to what ever addresses. That would be the AI that kills everyone. Including the AI itself in it’s sim would serve little functional role, and this AI won’t wirehead. It’s clear that the AGI risk is not about this.
edit: and to clarify, the problem with vague handwaving is that without defining what you handwave around, it is easy to produce stuff that is irrelevant, but appears relevant and math-y.
edit: hmm, seems that post with the virus making AI example didn’t get posted. Still, http://lesswrong.com/lw/bfj/evidence_for_the_orthogonality_thesis/68cf and http://lesswrong.com/lw/bfj/evidence_for_the_orthogonality_thesis/68eo convey the point. I’ve never said it is literally impossible to make a narrow AI that is rigged to tile the game world with blocks. It is, clearly, possible. One could make a glider gun iterator that finds the self replicating glider gun in the simulator, then some simple mechanisms set to make that gun in the real world. That is not a case of AI wanting to do something to the real world. That’s a glorified case of ‘my thermostat doesn’t wirehead’, to borrow from Will_Newsome.
Other issue is that one could immediately define some specific goal like ‘number of live cells’, and we could discuss this more specifically, instead of vague handwave about ill defined goal. But I can’t just define things narrowly for the other side of an argument. The wireheading is a problem of systems that can improve themselves. A system that can e.g. decide that it can’t figure out how to maximize live cells but it can prove some good theorems about four blocks.