I thought about the “burdensome details” objection some more and realized that I don’t understand it. Do you think the rescue sim idea would work? If yes, the FAI should either use it to rescue us, or find another course of action that’s even better—but either way we’d be saved from harm, no? If the FAI sees a child on a train track, believing that the FAI will somehow rescue it isn’t “burdensome detail”! So you should either believe that you’ll be rescued, or believe that rescue sims and other similar scenarios don’t work, or believe that we won’t create FAI.
The plan that’s even better won’t be about “rescuing the child” in particular, and for the same reason you can’t issue specific wishes to FAI, like to revive the cryopreserved.
But whatever the “better plan” might be, we know the FAI won’t leave the child there to die a horrible death. To borrow Eliezer’s analogy, I don’t know which moves Kasparov will make, but I do know he will win.
It’s not a given that rescuing the child is the best use of one’s resources. As a matter of heuristic, you’d expect that, and as a human, you’d form that particular wish, but it’s not obvious that even such heuristics will hold. Maybe something even better than rescuing the child can be done instead.
Not to speak of the situation where the harm is already done. Fact is a fact, not even a superintelligence can alter a fact. An agent determines, but doesn’t change. It could try “writing over” the tragedy with simulations of happy resolutions (in the future or rented possible worlds), but those simulations would be additional things to do, and not at all obviously optimal use of FAI’s control.
You’d expect the simularity of original scenario to “connect” the original scenario with the new ones, diluting the tradegy through reduction in anticipated experience of it happening, but anticipated experience has no absolute moral value, apart from allowing to discover moral value of certain facts. So this doesn’t even avert the tragedy, and simulation of sub-optimal pre-singularity world, even without the tragedy, even locally around the averted tragedy, might be grossly noneudaimonic.
But whatever the “better plan” might be, we know the FAI won’t leave the child there to die a horrible death. To borrow Eliezer’s analogy, I don’t know which moves Kasparov will make, but I do know he will win.
If that actually happened, it can’t be changed. An agent determines, never changes. Fact is a fact. And writing saved child “over” the fact of the actually harmed one, in future simulations or rented possible worlds, isn’t necessarily the best use of FAI’s control. So the best plan might well involve leaving that single fact be, with nothing done specifically “about” that situation.
I thought about the “burdensome details” objection some more and realized that I don’t understand it. Do you think the rescue sim idea would work? If yes, the FAI should either use it to rescue us, or find another course of action that’s even better—but either way we’d be saved from harm, no? If the FAI sees a child on a train track, believing that the FAI will somehow rescue it isn’t “burdensome detail”! So you should either believe that you’ll be rescued, or believe that rescue sims and other similar scenarios don’t work, or believe that we won’t create FAI.
The plan that’s even better won’t be about “rescuing the child” in particular, and for the same reason you can’t issue specific wishes to FAI, like to revive the cryopreserved.
But whatever the “better plan” might be, we know the FAI won’t leave the child there to die a horrible death. To borrow Eliezer’s analogy, I don’t know which moves Kasparov will make, but I do know he will win.
It’s not a given that rescuing the child is the best use of one’s resources. As a matter of heuristic, you’d expect that, and as a human, you’d form that particular wish, but it’s not obvious that even such heuristics will hold. Maybe something even better than rescuing the child can be done instead.
Not to speak of the situation where the harm is already done. Fact is a fact, not even a superintelligence can alter a fact. An agent determines, but doesn’t change. It could try “writing over” the tragedy with simulations of happy resolutions (in the future or rented possible worlds), but those simulations would be additional things to do, and not at all obviously optimal use of FAI’s control.
You’d expect the simularity of original scenario to “connect” the original scenario with the new ones, diluting the tradegy through reduction in anticipated experience of it happening, but anticipated experience has no absolute moral value, apart from allowing to discover moral value of certain facts. So this doesn’t even avert the tragedy, and simulation of sub-optimal pre-singularity world, even without the tragedy, even locally around the averted tragedy, might be grossly noneudaimonic.
If that actually happened, it can’t be changed. An agent determines, never changes. Fact is a fact. And writing saved child “over” the fact of the actually harmed one, in future simulations or rented possible worlds, isn’t necessarily the best use of FAI’s control. So the best plan might well involve leaving that single fact be, with nothing done specifically “about” that situation.