humans probably don’t even have a utility-function because our goals are not stable
Even what a human is is not stable.
My skin regenerates...my teeth are mineralized...I have some fillings...I eat some grapes...and spit out the pits...pull on some pants...and go drive a car. How am I qualitatively non-arbitrarily not my car?
The Voyager Horcrux has naked humans, but if hermit crabs were intelligent, would they send the probe up with an image of hermit crabs without shells?
Even what a human is is not stable.
My skin regenerates...my teeth are mineralized...I have some fillings...I eat some grapes...and spit out the pits...pull on some pants...and go drive a car. How am I qualitatively non-arbitrarily not my car?
The Voyager Horcrux has naked humans, but if hermit crabs were intelligent, would they send the probe up with an image of hermit crabs without shells?
I am not sure.