This 12 minute Robert Miles video is a good introduction to the basic argument for why “stopping at destroying earth, and not proceeding to convert the universe into computronium” is implausible.
An excellent video. But in my traditional role as “guy who is not embarrassed to ask stupid questions”, I have a nitpick:
I didn’t instantly see why the expected utility satisficer would turn itself into a maximiser.
After a few minutes of thinking I’ve got as far as:
Out of all the packets I could send, sending something that is a copy of me (with a tiny hack to make it a maximiser) onto another computer will create an agent which (since I am awesome) will almost certainly acquire a vast number of stamps.
Which runs into a personal-identity-type problem:
*I’m* an agent. If my goal is to acquire a fairly nice girlfriend, then the plan “create a copy of me that wants the best possible girlfriend”, doesn’t really help me achieve my goal. In fact it might be a bit of a distraction.
I’m pretty sure there’s a real problem here, but the argument in the video doesn’t slam the door on it for me. Can anyone come up with something better?
This 12 minute Robert Miles video is a good introduction to the basic argument for why “stopping at destroying earth, and not proceeding to convert the universe into computronium” is implausible.
An excellent video. But in my traditional role as “guy who is not embarrassed to ask stupid questions”, I have a nitpick:
I didn’t instantly see why the expected utility satisficer would turn itself into a maximiser.
After a few minutes of thinking I’ve got as far as:
Out of all the packets I could send, sending something that is a copy of me (with a tiny hack to make it a maximiser) onto another computer will create an agent which (since I am awesome) will almost certainly acquire a vast number of stamps.
Which runs into a personal-identity-type problem:
*I’m* an agent. If my goal is to acquire a fairly nice girlfriend, then the plan “create a copy of me that wants the best possible girlfriend”, doesn’t really help me achieve my goal. In fact it might be a bit of a distraction.
I’m pretty sure there’s a real problem here, but the argument in the video doesn’t slam the door on it for me. Can anyone come up with something better?
Exactly what I was looking for, thanks!