This is a possible AGI scenario, but it’s not clear why it should be particularly likely. For instance the AGI may reason that going aggressive will also be the fastest route to be terminated. Or the AGI may consider that keeping humans alive is good, since they were responsable for the AGI creation in the first place.
What you describe is the paper-clip maximiser scenario, which is arguably the most extreme end of the spectrum of super-AGI behaviours.
For instance the AGI may reason that going aggressive will also be the fastest route to be terminated
Absolutely! It may want to go aggressive, but reason that its best plan is to play nice until it can get into a position of strength.
What you describe is the paper-clip maximiser scenario, which is arguably the most extreme end of the spectrum of super-AGI behaviours.
So, in a sense, all rational agents are paperclip maximisers. Even the hoped-for ‘friendly AI’ is trying to get the most it can of what it wants, its just that what it wants is also what we want.
The striking thing about a paperclipper in particular is the simplicity of what it wants. But even an agent that has complex desires is in some sense trying to get the best score it can, as surely as it can.
This is a possible AGI scenario, but it’s not clear why it should be particularly likely. For instance the AGI may reason that going aggressive will also be the fastest route to be terminated. Or the AGI may consider that keeping humans alive is good, since they were responsable for the AGI creation in the first place.
What you describe is the paper-clip maximiser scenario, which is arguably the most extreme end of the spectrum of super-AGI behaviours.
Absolutely! It may want to go aggressive, but reason that its best plan is to play nice until it can get into a position of strength.
So, in a sense, all rational agents are paperclip maximisers. Even the hoped-for ‘friendly AI’ is trying to get the most it can of what it wants, its just that what it wants is also what we want.
The striking thing about a paperclipper in particular is the simplicity of what it wants. But even an agent that has complex desires is in some sense trying to get the best score it can, as surely as it can.