Why do you care if you’re running in a simulation? Well, if you’re a piece of malware, like a computer virus or Internet worm, you care whether you’re being analyzed by an anti-malware researcher. If you’re running in a researcher’s lab instead of actually infecting target systems, you’d want to shut down so the researcher can’t observe your real behavior and find out how to exterminate you. However, this means that if you can be fooled into believing you’re running in a simulation when you’re not, you can be made to shut down in the wild.
The WannaCrypt worm contained logic that basically said, “Look for the following properties in the Internet. If you observe them, that means you’re not running in the real Internet; you’re running in a simulation.” But the researcher was able to cause those properties to become true in the real Internet, thereby convincing the live malware that was infesting the actual Internet to believe it was in a simulation and shut down.
Anti-analysis or anti-debugging features, which attempt to ask “Am I running in a simulation?”, are not a new thing in malware, or in other programs that attempt to extract value from humans — such as copy-protection routines. But they do make malware an interesting example of a type of agent for which the simulation hypothesis matters, and where mistaken beliefs about whether you’re in a simulation can have devastating effects on your ability to function.
Why do you think a piece of software has the same goals as it’s creator? My conscious planning mind doesn’t have the same goals as evolution.
Current software doesn’t even have goals, it has behaviors. Ascribing desires and decision-making to it leads to incorrect beliefs. AIs will have goals, but they’ll be influenced and shaped by their creators rather than being fully specified.
Why do you care if you’re running in a simulation? Well, if you’re a piece of malware, like a computer virus or Internet worm, you care whether you’re being analyzed by an anti-malware researcher. If you’re running in a researcher’s lab instead of actually infecting target systems, you’d want to shut down so the researcher can’t observe your real behavior and find out how to exterminate you. However, this means that if you can be fooled into believing you’re running in a simulation when you’re not, you can be made to shut down in the wild.
Yep, that happened. It’s not even the first time.
The WannaCrypt worm contained logic that basically said, “Look for the following properties in the Internet. If you observe them, that means you’re not running in the real Internet; you’re running in a simulation.” But the researcher was able to cause those properties to become true in the real Internet, thereby convincing the live malware that was infesting the actual Internet to believe it was in a simulation and shut down.
Anti-analysis or anti-debugging features, which attempt to ask “Am I running in a simulation?”, are not a new thing in malware, or in other programs that attempt to extract value from humans — such as copy-protection routines. But they do make malware an interesting example of a type of agent for which the simulation hypothesis matters, and where mistaken beliefs about whether you’re in a simulation can have devastating effects on your ability to function.
Why do you think a piece of software has the same goals as it’s creator? My conscious planning mind doesn’t have the same goals as evolution.
Current software doesn’t even have goals, it has behaviors. Ascribing desires and decision-making to it leads to incorrect beliefs. AIs will have goals, but they’ll be influenced and shaped by their creators rather than being fully specified.