Here’s another, unpolitical scenario about pulling back values
Consider a world where there are cosmic rays that can hit an agent’s brain, but they home in on the part containing the utility function. Shielding from these rays is possible but expensive.
In this world, when an agent considers whether to invest in the expensive protection, it considers whether a version of it newly hit with a cosmic ray would remain loyal to the old version of it by keeping on maximizing the old utility function (with some weight) as well as the new cosmic ray begotten one.
Then, when a agent newly hit by a cosmic ray agent considers whether to remain loyal, it notices that if it doesn’t, it’s less likely to exist since it’s predecessor would have invested in the protection, so it remains loyal.
Interesting, thanks! I can’t formalize your scenario because I don’t understand it completely, but it looks like a game theory problem that should yield an equilibrium in mixed strategies, not unconditional loyalty.
This wasn’t about people but generic game-theoretic agents (and all else equal generic game-theoretic agents prefer to exist because then there will be someone in the world with their utility function exerting an influence on the world so as to make it rate higher in their utility function than it would have if there wasn’t anyone).
Here’s another, unpolitical scenario about pulling back values
Interesting, thanks! I can’t formalize your scenario because I don’t understand it completely, but it looks like a game theory problem that should yield an equilibrium in mixed strategies, not unconditional loyalty.
This assumes that the new agent prefers to have existed, and it’s not clear to me that people ordinarily have such a preference.
This wasn’t about people but generic game-theoretic agents (and all else equal generic game-theoretic agents prefer to exist because then there will be someone in the world with their utility function exerting an influence on the world so as to make it rate higher in their utility function than it would have if there wasn’t anyone).
Ah, good point.