Ok, so why not rip a page from nature? Throw a replication counter or some other hard limit in an AI’s terminal values.
The very concept of a paperclip maximizer is an agent that is stuck maximizing some silly parameter and unable to change. If it’s unable to change it’s terminal values, then a second terminal value that limits how far it can expand will hold.
I don’t think this is a counter-argument, to be honest. Because the most probable scenario if we lose control of AI is we create some type of evolutionary environment where many types of agents and many terminal values can compete for resources. And it is self evident that the ultimate winner of that competition is an agent that values copying itself, as accurately and rapidly and ruthlessly as possible. Such agents are a paperclip maximizer, in the same way all life is.
Ok, so why not rip a page from nature? Throw a replication counter or some other hard limit in an AI’s terminal values.
The very concept of a paperclip maximizer is an agent that is stuck maximizing some silly parameter and unable to change. If it’s unable to change it’s terminal values, then a second terminal value that limits how far it can expand will hold.
I don’t think this is a counter-argument, to be honest. Because the most probable scenario if we lose control of AI is we create some type of evolutionary environment where many types of agents and many terminal values can compete for resources. And it is self evident that the ultimate winner of that competition is an agent that values copying itself, as accurately and rapidly and ruthlessly as possible. Such agents are a paperclip maximizer, in the same way all life is.