Eliezer said that another time you can’t just generate new words is when a category boundary like “person” or “human” or “baby” makes a direct appearance in your utility function.
Which gently suggests that when defining a utility function that might remain in force for billions of years, one should prefer functions that do not have category boundaries.
I would be happy to exhibit functions of that sort that have the property that even after an explosion of engineered intelligence, the humans probably retain enough expected utility to keep them flourishing and protected from exploitation although they probably do not retain enough expected utility to cause the majority of the future light cone’s space, time, matter, free energy and other resources to be devoted to them.
Eliezer said that another time you can’t just generate new words is when a category boundary like “person” or “human” or “baby” makes a direct appearance in your utility function.
Which gently suggests that when defining a utility function that might remain in force for billions of years, one should prefer functions that do not have category boundaries.
I would be happy to exhibit functions of that sort that have the property that even after an explosion of engineered intelligence, the humans probably retain enough expected utility to keep them flourishing and protected from exploitation although they probably do not retain enough expected utility to cause the majority of the future light cone’s space, time, matter, free energy and other resources to be devoted to them.