“human values” being maximized by a Singleton forever would importantly fall short of my ideal future
I would expect that letting (other) people define themselves is part of “human values”, and so maximizing influence of such values on the world would let decisions of individual existing people screen off Singleton’s decisions at least when it comes to their own development. Any decision of a Singleton about how a person’s thinking and values should be developing is not legitimate to that person’s values if it doesn’t ultimately follow that person’s own decisions in some way. Values don’t define preference over just the end states of a world, they define how the initial conditions that are already in place should develop, and existing people are part of the initial conditions.
This works even if a Singleton literally writes down all of the future, including people and their thoughts, in the same way as this goes with physics writing down the future. Decisions of people embedded in a Singleton can still remain their own, the same as with people embedded in physics, it’s just another setting for making decisions within a lawful environment/substrate.
I would expect that letting (other) people define themselves is part of “human values”, and so maximizing influence of such values on the world would let decisions of individual existing people screen off Singleton’s decisions at least when it comes to their own development. Any decision of a Singleton about how a person’s thinking and values should be developing is not legitimate to that person’s values if it doesn’t ultimately follow that person’s own decisions in some way. Values don’t define preference over just the end states of a world, they define how the initial conditions that are already in place should develop, and existing people are part of the initial conditions.
This works even if a Singleton literally writes down all of the future, including people and their thoughts, in the same way as this goes with physics writing down the future. Decisions of people embedded in a Singleton can still remain their own, the same as with people embedded in physics, it’s just another setting for making decisions within a lawful environment/substrate.