I assume, on this basis, that you agree with the hypothetical at the end?
Nope. I still don’t agree that simulating an agent N times and doing X to one of them is morally equivalent to risking X to them with probability 1/(N+1). For example, if you are not at all altruistic to copies of yourself, then you don’t care about the former situation as long as the copy that X is being done to is not you. On the other hand, if you value fairness among your copies (that is, if you value your copies having similar quality of life) then you care about the former situation more strongly than the latter situation.
By this logic, the moment you turn on a provably Friendly AI you should destroy it, because it might have hacked you into thinking it’s friendly. Worse still, a hostile god would presumably realize you wont believe it, and so hack you into thinking it’s not godlike; so anything claiming not to be godlike is lying.
Pretty much the only thing a godlike agent can convince me of is that it’s godlike (and I am not even totally convinced this is possible). After that, again, whatever evidence a godlike agent presents of anything else could have been fabricated. Your last inference doesn’t follow from the others; my priors regarding the prevalence of godlike agents is currently extremely low, and claiming not to be godlike is not strong evidence either way.
To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn’t it be equivalent to Alice-who-values-copies-of-herself?
my priors regarding the prevalence of godlike agents is currently extremely low,
And what are those priors based on? Evidence! Evidence that a godlike being would be motivated to falsify!
To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn’t it be equivalent to Alice-who-values-copies-of-herself?
Sure, but the result you describe is equivalent to Alice being an average utilitarian with respect to copies of herself. What if Alice is a total utilitarian with respect to copies of herself?
Nope. I still don’t agree that simulating an agent N times and doing X to one of them is morally equivalent to risking X to them with probability 1/(N+1). For example, if you are not at all altruistic to copies of yourself, then you don’t care about the former situation as long as the copy that X is being done to is not you. On the other hand, if you value fairness among your copies (that is, if you value your copies having similar quality of life) then you care about the former situation more strongly than the latter situation.
Pretty much the only thing a godlike agent can convince me of is that it’s godlike (and I am not even totally convinced this is possible). After that, again, whatever evidence a godlike agent presents of anything else could have been fabricated. Your last inference doesn’t follow from the others; my priors regarding the prevalence of godlike agents is currently extremely low, and claiming not to be godlike is not strong evidence either way.
To be clear, since humans are specified as valuing all agents (including sims of themselves and others) shouldn’t it be equivalent to Alice-who-values-copies-of-herself?
And what are those priors based on? Evidence! Evidence that a godlike being would be motivated to falsify!
Sure, but the result you describe is equivalent to Alice being an average utilitarian with respect to copies of herself. What if Alice is a total utilitarian with respect to copies of herself?
Actually, she should still make the same choice, although she would choose differently in other scenarios.