I think this becomes clearer if we distinguish people from agents. People are somewhat agentic beings of high moral value, while dedicated agents might have little moral value. Maximizing agency of people too well probably starts destroying their value at some point. At present, getting as much agency as possible out of potentially effective people is important for instrumental reasons, since only humans can be agentic, but that will change.
It’s useful for agents to have legible values, so that they can build complicated systems that serve much simpler objectives well. But for people it’s less obviously important to have much clarity to their values, especially if they are living in a world managed by agents. Agents managing the world do need clear understanding of values of civilization, but even then it doesn’t necessarily make sense to compare these values with those of individual people.
(It’s not completely obvious that individual people are one of the most valuable things to enact, so a well-developed future might lack them. Even if values of civilization are determined by people actually living through eons of reflection and change, and not by a significantly less concrete process, that gives enough distance from the present perspective to doubt anything about the result.)
I think this becomes clearer if we distinguish people from agents. People are somewhat agentic beings of high moral value, while dedicated agents might have little moral value. Maximizing agency of people too well probably starts destroying their value at some point. At present, getting as much agency as possible out of potentially effective people is important for instrumental reasons, since only humans can be agentic, but that will change.
It’s useful for agents to have legible values, so that they can build complicated systems that serve much simpler objectives well. But for people it’s less obviously important to have much clarity to their values, especially if they are living in a world managed by agents. Agents managing the world do need clear understanding of values of civilization, but even then it doesn’t necessarily make sense to compare these values with those of individual people.
(It’s not completely obvious that individual people are one of the most valuable things to enact, so a well-developed future might lack them. Even if values of civilization are determined by people actually living through eons of reflection and change, and not by a significantly less concrete process, that gives enough distance from the present perspective to doubt anything about the result.)