As has already been said by many: an optimizing system will optimize for whatever it is designed to optimize for.
If it is designed to optimize for maximizing the value of a counter in a register somewhere, it will do that… maybe find INT_MAX and halt, maybe find some way of rearchitecting itself so that INT_MAX is incremented, depending on how powerful an optimizer it is. It will not spontaneously decide to start making paperclips or making happy people or sorting pebbles into prime-numbered piles instead.
Conversely, if it is designed to optimize for maximizing the number of paperclips, it will do that—and will not spontaneously decide to start maximizing the value of a counter in a register somewhere.
I think the reason this is sometimes confusing is that people confuse the functional description with the structural one. Sure, maybe the implementation of the paperclip-optimizer involves maximizing counters in registers, but as long as it actually is a paperclip-optimizer it has no reason to further optimize for counter-maximization beyond whatever such optimization is designed into its architecture. (Indeed, it might replace its architecture with a superior paperclip-optimization implementation that doesn’t involve counters, as an anti-akrasia method.)
As has already been said by many: an optimizing system will optimize for whatever it is designed to optimize for.
If it is designed to optimize for maximizing the value of a counter in a register somewhere, it will do that… maybe find INT_MAX and halt, maybe find some way of rearchitecting itself so that INT_MAX is incremented, depending on how powerful an optimizer it is. It will not spontaneously decide to start making paperclips or making happy people or sorting pebbles into prime-numbered piles instead.
Conversely, if it is designed to optimize for maximizing the number of paperclips, it will do that—and will not spontaneously decide to start maximizing the value of a counter in a register somewhere.
I think the reason this is sometimes confusing is that people confuse the functional description with the structural one. Sure, maybe the implementation of the paperclip-optimizer involves maximizing counters in registers, but as long as it actually is a paperclip-optimizer it has no reason to further optimize for counter-maximization beyond whatever such optimization is designed into its architecture. (Indeed, it might replace its architecture with a superior paperclip-optimization implementation that doesn’t involve counters, as an anti-akrasia method.)