But I assume a utility function maximizer doesn’t draw a qualitative difference between these two categories—it’s just a difference in magnitude.
When the agents are choosing between “live and let live” and “a total war”, the war involves a loss of resources. The line I was hinting as was something like: suppose we can’t conquer those planets, but we could destroy them—should we?
From humanity’s perspective, destroying a planet is not an improvement over having it converted to paperclips. But if there is some kind of Hell machine, building gigantic eternal torture chambers, we might prefer those torture chambers to be destroyed.
From the paperclip maximizer’s perspective, I guess, if it’s not paperclips, it does not make a difference.
there’s no reason they can’t just beam hostile radio signals at each other all day long.
Signals are useless unless received. A planet governed by an AI could adopt a “don’t look up” policy.
Perhaps there will be a technology to create some fog between solar systems that would stop the signals. Though that’s dangerous: if you don’t see the signals, you also don’t see potential spaceships flying towards you.
I guess an AI could implement some kind of firewall, make a smaller AI that only observes the part of the universe, reports spaceships and ignores everything else. But then the obvious response would be to send the spaceships along with the signal. Then, destroying things along the boundary becomes relevant again.
I guess there might be some interesting arms race about how much toxic information can you include in something that the other side cannot ignore, such as the structure of your attacking spaceships. I imagine something like sending million spaceships that can physically attack the enemy bases, but also their positions encode some infohazard, so when the enemy starts monitoring them, it inevitably works with the dangerous data. (As a silly example, imagine a fleet of spaceships flying in a formation that spells “this sentence is false”.)
When the agents are choosing between “live and let live” and “a total war”, the war involves a loss of resources. The line I was hinting as was something like: suppose we can’t conquer those planets, but we could destroy them—should we?
From humanity’s perspective, destroying a planet is not an improvement over having it converted to paperclips. But if there is some kind of Hell machine, building gigantic eternal torture chambers, we might prefer those torture chambers to be destroyed.
From the paperclip maximizer’s perspective, I guess, if it’s not paperclips, it does not make a difference.
Signals are useless unless received. A planet governed by an AI could adopt a “don’t look up” policy.
Perhaps there will be a technology to create some fog between solar systems that would stop the signals. Though that’s dangerous: if you don’t see the signals, you also don’t see potential spaceships flying towards you.
I guess an AI could implement some kind of firewall, make a smaller AI that only observes the part of the universe, reports spaceships and ignores everything else. But then the obvious response would be to send the spaceships along with the signal. Then, destroying things along the boundary becomes relevant again.
I guess there might be some interesting arms race about how much toxic information can you include in something that the other side cannot ignore, such as the structure of your attacking spaceships. I imagine something like sending million spaceships that can physically attack the enemy bases, but also their positions encode some infohazard, so when the enemy starts monitoring them, it inevitably works with the dangerous data. (As a silly example, imagine a fleet of spaceships flying in a formation that spells “this sentence is false”.)