It’s not conclusive, I don’t have some important parts of the puzzle yet. The question is what makes some systems invasive and others not, why a PC with a complicated algorithm that outputs originally unknown results with known properties (that would qualify as a narrow target) is as dangerous as a rock, but some kinds of AI will try to compute outside the box. My best semitechnical guess is that it has something to do with AI having a level of modeling the world that allows the system to view the substrate on which it executes and the environment outside the box as being involved in the same computational process, so that following the algorithm inside the box becomes a special case of computing on the physical substrate outside the box (and computing on the physical substrate means determining the state of the physical world, with building physical structures as a special case). Which, if not explicitly prohibited, might be more efficient for whatever goal is specified, even if this goal is supposed to be realized inside the box (inside the future of the box).
Shane: Re dangerous GZIP.
It’s not conclusive, I don’t have some important parts of the puzzle yet. The question is what makes some systems invasive and others not, why a PC with a complicated algorithm that outputs originally unknown results with known properties (that would qualify as a narrow target) is as dangerous as a rock, but some kinds of AI will try to compute outside the box. My best semitechnical guess is that it has something to do with AI having a level of modeling the world that allows the system to view the substrate on which it executes and the environment outside the box as being involved in the same computational process, so that following the algorithm inside the box becomes a special case of computing on the physical substrate outside the box (and computing on the physical substrate means determining the state of the physical world, with building physical structures as a special case). Which, if not explicitly prohibited, might be more efficient for whatever goal is specified, even if this goal is supposed to be realized inside the box (inside the future of the box).