Indeed—this is very likely a crux. I’d enjoy it if other humans obeyed my stated preferences, but I think I’d lose respect for them as agents (and making very specific object-level requests would show my disrespect for them as moral targets).
Doing things that I project to improve overall quality of life for many people is good, IMO. Following arbitrary stated preferences is very rarely an effective way to do that. There are lots of cases where statements are a good hint to utility weightings, and lots of cases where the speaker is confused or misleading or time-inconsistent.
Dead people’s historical statements, always, are incorrect about what will improve their experienced universe.
Indeed—this is very likely a crux. I’d enjoy it if other humans obeyed my stated preferences, but I think I’d lose respect for them as agents (and making very specific object-level requests would show my disrespect for them as moral targets).
Doing things that I project to improve overall quality of life for many people is good, IMO. Following arbitrary stated preferences is very rarely an effective way to do that. There are lots of cases where statements are a good hint to utility weightings, and lots of cases where the speaker is confused or misleading or time-inconsistent.
Dead people’s historical statements, always, are incorrect about what will improve their experienced universe.