Nonwantons, however, can represent a model of an idealized preference structure — perhaps, for example, a model based on a superordinate judgment of long-term lifespan considerations… So a human can say: I would prefer to prefer not to smoke. This second-order preference can then become a motivational competitor to the first-order preference. At the level of second-order preferences, I prefer to prefer to not smoke; nevertheless, as a first-order preference, I prefer to smoke.
One problem: How do we distinguish actual second-order preferences (“I would prefer to prefer not to smoke”) from improper beliefs about one’s own preferences, e.g. belief in belief (“It is good to think that smoking is bad”)?
It seems to me that the obvious answer is to ask, “Well, is smoking actually bad?” In other words, we shouldn’t expect to find out how good our reflective preferences are without actually asking what sort of world we live in, and whether agents with those preferences tend to do well in that sort of world.
One problem: How do we distinguish actual second-order preferences (“I would prefer to prefer not to smoke”) from improper beliefs about one’s own preferences, e.g. belief in belief (“It is good to think that smoking is bad”)?
It seems to me that the obvious answer is to ask, “Well, is smoking actually bad?” In other words, we shouldn’t expect to find out how good our reflective preferences are without actually asking what sort of world we live in, and whether agents with those preferences tend to do well in that sort of world.
“Actually bad” and “do well” depend on values, right? So that seems like the start to a better approach, but isn’t enough.