Let it build a machine that takes input from own surroundings.
But the new machine can’t self-modify. My point is about the limitations of cousin_it’s example. The machine has a completely accurate model of the world as input and uses an extremely inefficient algorithm to find a way to paperclip the world.
The second machine can be designed to build a third machine, based on the second machine’s observations.
Yes, but now the argument that you will converge to a paper clipper is much weaker.
Let it build a machine that takes input from own surroundings.
But the new machine can’t self-modify. My point is about the limitations of cousin_it’s example. The machine has a completely accurate model of the world as input and uses an extremely inefficient algorithm to find a way to paperclip the world.
The second machine can be designed to build a third machine, based on the second machine’s observations.
Yes, but now the argument that you will converge to a paper clipper is much weaker.