Your scenario is burdened by excessive detail about FAI. Any situation in which people create lots of sims but don’t allow lots of suffering/horror in the sims (perhaps as “rescue sims,” perhaps because of something like animal welfare laws, or many other possibilities) poses almost the same questions.
I thought about the “burdensome details” objection some more and realized that I don’t understand it. Do you think the rescue sim idea would work? If yes, the FAI should either use it to rescue us, or find another course of action that’s even better—but either way we’d be saved from harm, no? If the FAI sees a child on a train track, believing that the FAI will somehow rescue it isn’t “burdensome detail”! So you should either believe that you’ll be rescued, or believe that rescue sims and other similar scenarios don’t work, or believe that we won’t create FAI.
The plan that’s even better won’t be about “rescuing the child” in particular, and for the same reason you can’t issue specific wishes to FAI, like to revive the cryopreserved.
But whatever the “better plan” might be, we know the FAI won’t leave the child there to die a horrible death. To borrow Eliezer’s analogy, I don’t know which moves Kasparov will make, but I do know he will win.
It’s not a given that rescuing the child is the best use of one’s resources. As a matter of heuristic, you’d expect that, and as a human, you’d form that particular wish, but it’s not obvious that even such heuristics will hold. Maybe something even better than rescuing the child can be done instead.
Not to speak of the situation where the harm is already done. Fact is a fact, not even a superintelligence can alter a fact. An agent determines, but doesn’t change. It could try “writing over” the tragedy with simulations of happy resolutions (in the future or rented possible worlds), but those simulations would be additional things to do, and not at all obviously optimal use of FAI’s control.
You’d expect the simularity of original scenario to “connect” the original scenario with the new ones, diluting the tradegy through reduction in anticipated experience of it happening, but anticipated experience has no absolute moral value, apart from allowing to discover moral value of certain facts. So this doesn’t even avert the tragedy, and simulation of sub-optimal pre-singularity world, even without the tragedy, even locally around the averted tragedy, might be grossly noneudaimonic.
But whatever the “better plan” might be, we know the FAI won’t leave the child there to die a horrible death. To borrow Eliezer’s analogy, I don’t know which moves Kasparov will make, but I do know he will win.
If that actually happened, it can’t be changed. An agent determines, never changes. Fact is a fact. And writing saved child “over” the fact of the actually harmed one, in future simulations or rented possible worlds, isn’t necessarily the best use of FAI’s control. So the best plan might well involve leaving that single fact be, with nothing done specifically “about” that situation.
I don’t say rescue sims are strictly impossible in the above argument, indeed I said that everything is possible (in the sense of being in the domain of prior, roughly speaking), but you anticipate only a tiny fraction of what’s possible (or likely), and rescue sims probably don’t fall into that area. I agree with Carl that your FAI scenario is unlikely to the point of impossible (in the sense of prior, not just anticipation).
That would fall under “nitpicking”. When I said “impossible” I meant to say “they won’t work on us here”. Or will work with negligible probability, which is pretty much the same thing. My question to Carl stands: does he agree that it’s impossible/pointless to save people in the past by building rescue sims? Is this a consequence of UDT, the way he understands it?
A word on nitpicking: even if I believe it’s likely you meant a given thing, if it’s nonetheless not clear that you didn’t mean another, or presentation doesn’t make it clear for other people that you didn’t mean another, it’s still better to debias the discussion from illusion of transparency by explicitly disambiguating than relying on fitting the words to a model that was never explicitly tested.
There is an essential ambiguity for this discussion between “pointless” because subjective anticipation won’t allow you noticing, and “pointless” because it doesn’t optimize goodness as well as other plans do. It might be pointless saving people in the past by building sims, but probably only for the same reason it might be pointless reviving the cryonauts: because there are even better decisions available.
Your scenario is burdened by excessive detail about FAI. Any situation in which people create lots of sims but don’t allow lots of suffering/horror in the sims (perhaps as “rescue sims,” perhaps because of something like animal welfare laws, or many other possibilities) poses almost the same questions.
I thought about the “burdensome details” objection some more and realized that I don’t understand it. Do you think the rescue sim idea would work? If yes, the FAI should either use it to rescue us, or find another course of action that’s even better—but either way we’d be saved from harm, no? If the FAI sees a child on a train track, believing that the FAI will somehow rescue it isn’t “burdensome detail”! So you should either believe that you’ll be rescued, or believe that rescue sims and other similar scenarios don’t work, or believe that we won’t create FAI.
The plan that’s even better won’t be about “rescuing the child” in particular, and for the same reason you can’t issue specific wishes to FAI, like to revive the cryopreserved.
But whatever the “better plan” might be, we know the FAI won’t leave the child there to die a horrible death. To borrow Eliezer’s analogy, I don’t know which moves Kasparov will make, but I do know he will win.
It’s not a given that rescuing the child is the best use of one’s resources. As a matter of heuristic, you’d expect that, and as a human, you’d form that particular wish, but it’s not obvious that even such heuristics will hold. Maybe something even better than rescuing the child can be done instead.
Not to speak of the situation where the harm is already done. Fact is a fact, not even a superintelligence can alter a fact. An agent determines, but doesn’t change. It could try “writing over” the tragedy with simulations of happy resolutions (in the future or rented possible worlds), but those simulations would be additional things to do, and not at all obviously optimal use of FAI’s control.
You’d expect the simularity of original scenario to “connect” the original scenario with the new ones, diluting the tradegy through reduction in anticipated experience of it happening, but anticipated experience has no absolute moral value, apart from allowing to discover moral value of certain facts. So this doesn’t even avert the tragedy, and simulation of sub-optimal pre-singularity world, even without the tragedy, even locally around the averted tragedy, might be grossly noneudaimonic.
If that actually happened, it can’t be changed. An agent determines, never changes. Fact is a fact. And writing saved child “over” the fact of the actually harmed one, in future simulations or rented possible worlds, isn’t necessarily the best use of FAI’s control. So the best plan might well involve leaving that single fact be, with nothing done specifically “about” that situation.
Nesov says rescue sims are impossible, while you only say my FAI scenario is unlikely. But you claim to be thinking about the same UDT. Why is that?
I don’t say rescue sims are strictly impossible in the above argument, indeed I said that everything is possible (in the sense of being in the domain of prior, roughly speaking), but you anticipate only a tiny fraction of what’s possible (or likely), and rescue sims probably don’t fall into that area. I agree with Carl that your FAI scenario is unlikely to the point of impossible (in the sense of prior, not just anticipation).
That would fall under “nitpicking”. When I said “impossible” I meant to say “they won’t work on us here”. Or will work with negligible probability, which is pretty much the same thing. My question to Carl stands: does he agree that it’s impossible/pointless to save people in the past by building rescue sims? Is this a consequence of UDT, the way he understands it?
A word on nitpicking: even if I believe it’s likely you meant a given thing, if it’s nonetheless not clear that you didn’t mean another, or presentation doesn’t make it clear for other people that you didn’t mean another, it’s still better to debias the discussion from illusion of transparency by explicitly disambiguating than relying on fitting the words to a model that was never explicitly tested.
There is an essential ambiguity for this discussion between “pointless” because subjective anticipation won’t allow you noticing, and “pointless” because it doesn’t optimize goodness as well as other plans do. It might be pointless saving people in the past by building sims, but probably only for the same reason it might be pointless reviving the cryonauts: because there are even better decisions available.
To clarify: I already accept the objections about “burdensome details” and “better plans”. I’m only interested in the subjective anticipation angle.
ETA: sometime after writing this comment I stopped understanding those objections, but anticipation still interests me more.