I agree with the above comments that concern for future individuals would be contained in the utility functions of people who exist now, but there’s an ambiguity in the AI’s utility function in that it seems forbidden to consider the future or past output of it’s utility function. By limiting itself to the concern of the people who currently exist, if it were to try and maximize this output over all time it would then be concerning itself with people who do not yet or no longer exist, which is at direct odds with its utility function. Being barred from such considerations, it could make sense to change it’s own utility function to restrict concern to the people existing at that tame, IF this is what most satisfied the preference of those people.
While the default near-sightedness of people is bad news here, if the AI succeeds in modelling us as “smarter, more the people we want to be” etc, then its utility function seems unlikely to become so fixed in time.
I agree with the above comments that concern for future individuals would be contained in the utility functions of people who exist now, but there’s an ambiguity in the AI’s utility function in that it seems forbidden to consider the future or past output of it’s utility function. By limiting itself to the concern of the people who currently exist, if it were to try and maximize this output over all time it would then be concerning itself with people who do not yet or no longer exist, which is at direct odds with its utility function. Being barred from such considerations, it could make sense to change it’s own utility function to restrict concern to the people existing at that tame, IF this is what most satisfied the preference of those people.
While the default near-sightedness of people is bad news here, if the AI succeeds in modelling us as “smarter, more the people we want to be” etc, then its utility function seems unlikely to become so fixed in time.