I would also add that people are often poorly calibrated on what “cheaply satisfied” means, and dismiss personal inconvenience or displeasure as not truly being costs. If the AI prefers that nobody in the world use the color violet, that’s not a cheaply satisfied preference, even if eliminating violet just means more personal inconvenience.
I say no. See https://www.lesswrong.com/posts/bWdtxtyTiEsxSZtK3/a-world-without-violet-peculiar-consequences-of-granting .
I would also add that people are often poorly calibrated on what “cheaply satisfied” means, and dismiss personal inconvenience or displeasure as not truly being costs. If the AI prefers that nobody in the world use the color violet, that’s not a cheaply satisfied preference, even if eliminating violet just means more personal inconvenience.