I think similar sentiments are largely a failure of the imagination. The possibilities demand a whole lot of imagination.
The only thing you can’t have post singularity is truly suffering people to help. And if you must have that and refuse to tweak your reward system so you don’t, you can enter a simulation where it seems exactly like you have that.
If you want a mundane existence you can simulate that until you’re bored, then join the crowd doing things that are really new and exciting.
You don’t stop being you from any little tweak to your reward system or memory. And they’re all reversible.
what’s the principle here? if an agent would have the same observations in world W and W’ then their preferences must be indifferent between W and W’ ? this seems clearly false.
I wouldn’t make that argument. I just don’t see the point of keeping it real.
It just seems like going virtual opens up a lot of possbilities with no downside. If you want consistency and real work, put it in your sim. Share it with other real people if you want to compromise on what worlds you’ll inhabit and what challenges you’ll face.
If you want real people who are really suffering so there are real stakes to play for, well, that’s an orthogonal issue. I’d rather see nonconsensual suffering eliminated.
So: Why prefer the Real? What’s it got that the Virtual doesn’t?
For example, the agent might decide that its utility function of anything that the agent knows to be virtual is close to zero because the agent believes in a real-world mission (e.g. Agent-2 was supposed to eventually reach the SC level and do actual AI-related research, but it was also trained to solve simulated long-term tasks like playing through video games)
As for reasons to believe that the contribution of anything virtual into the utility function is close to zero… one level is opportunity costs undermining real-world outcomes[1] in exchange for something useless (e.g. a schoolboy’s knowledge vs. missions passed in GTA). The next level is the reasons for real-world outcomes to be important. Before the possibility of a post-work future, society’s members were supposed to do work that others deem useful enough to pay for it, and it would somehow increase the well-being of the collective’s members or help the whole collective to reach its terminal goals (e.g. inspire its members to be more creative or work harder). The virtual world is known to be a superstimulus which could be as unlikely to increase the collective’s well-being as fast food causing people to become obese.
This is under the assumption that things have gone well, so everyone is very empowered to have the life they want, and the assumption of benevolent superintelligence that can help better than any human if someone wants help.
I think similar sentiments are largely a failure of the imagination. The possibilities demand a whole lot of imagination.
The only thing you can’t have post singularity is truly suffering people to help. And if you must have that and refuse to tweak your reward system so you don’t, you can enter a simulation where it seems exactly like you have that.
If you want a mundane existence you can simulate that until you’re bored, then join the crowd doing things that are really new and exciting.
You don’t stop being you from any little tweak to your reward system or memory. And they’re all reversible.
Intuitions fail here for a good reason.
The possibilities are limitless.
My mundane values care about real physical stuff not simulated stuff.
If you can’t tell the difference, how could you care which is which?
I’m talking about blocking your memories of living in a simulated world.
what’s the principle here? if an agent would have the same observations in world W and W’ then their preferences must be indifferent between W and W’ ? this seems clearly false.
I wouldn’t make that argument. I just don’t see the point of keeping it real.
It just seems like going virtual opens up a lot of possbilities with no downside. If you want consistency and real work, put it in your sim. Share it with other real people if you want to compromise on what worlds you’ll inhabit and what challenges you’ll face.
If you want real people who are really suffering so there are real stakes to play for, well, that’s an orthogonal issue. I’d rather see nonconsensual suffering eliminated.
So: Why prefer the Real? What’s it got that the Virtual doesn’t?
For example, the agent might decide that its utility function of anything that the agent knows to be virtual is close to zero because the agent believes in a real-world mission (e.g. Agent-2 was supposed to eventually reach the SC level and do actual AI-related research, but it was also trained to solve simulated long-term tasks like playing through video games)
As for reasons to believe that the contribution of anything virtual into the utility function is close to zero… one level is opportunity costs undermining real-world outcomes[1] in exchange for something useless (e.g. a schoolboy’s knowledge vs. missions passed in GTA). The next level is the reasons for real-world outcomes to be important. Before the possibility of a post-work future, society’s members were supposed to do work that others deem useful enough to pay for it, and it would somehow increase the well-being of the collective’s members or help the whole collective to reach its terminal goals (e.g. inspire its members to be more creative or work harder). The virtual world is known to be a superstimulus which could be as unlikely to increase the collective’s well-being as fast food causing people to become obese.
Including things like actual skills learned during games, as happened with Agent-2′s ability to solve long-term tasks.
You’re assuming that there will be a distinction post-singularity? Are you not a physicalist?
Why do you say this?
This is under the assumption that things have gone well, so everyone is very empowered to have the life they want, and the assumption of benevolent superintelligence that can help better than any human if someone wants help.