That may be true too. It depends on the priors we have for generic superhuman agents’ reasons for keeping a simulation running (e.g., having some other science experiments planned, wanting to reward you for providing data...) vs. for shutting it down (e.g., vindictiveness, energy conservation, being interested only in one data point per simulation...).
We do have some data to work with here, since we have experience with the differential effects of power, intelligence, curiosity, etc. among humans. That data is only weakly applicable to such an exotic agent, but it does play a role, so our uncertainty isn’t absolute. My main point was that unusual situations like this don’t call for complete decision-theoretic despair; we still need to make choices, and we can still do so reasonably, though our confidence that the best decision is also a winning decision is greatly diminished.
That may be true too. It depends on the priors we have for generic superhuman agents’ reasons for keeping a simulation running (e.g., having some other science experiments planned, wanting to reward you for providing data...) vs. for shutting it down (e.g., vindictiveness, energy conservation, being interested only in one data point per simulation...).
We do have some data to work with here, since we have experience with the differential effects of power, intelligence, curiosity, etc. among humans. That data is only weakly applicable to such an exotic agent, but it does play a role, so our uncertainty isn’t absolute. My main point was that unusual situations like this don’t call for complete decision-theoretic despair; we still need to make choices, and we can still do so reasonably, though our confidence that the best decision is also a winning decision is greatly diminished.