Both you and Eliezer seem to be replying to this argument:
People only intrinsically desire pleasure.
An FAI should maximize whatever people intrinsically desire.
Therefore, an FAI should maximize pleasure.
I am convinced that this argument fails for the reasons you cite. But who is making that argument? Is this supposed to be the best argument for hedonistic utilitarianism?
A recent study by folks at the Oxford Centre for Neuroethics suggests that Greene et. al.’s results are better explained by appeal to differences in how intuitive/counterintuitive a moral judgment is, rather than differences in how utilitarian/deontological it is. I had a look at the study, and it seems reasonably legit, but I don’t have any expertise in neuroscience. As I understand it, their findings suggest that the “more cognitive” part of the brain gets recruited more when making a counterintuitive moral judgment, whether utilitarian or deontological.
Also, it is worth noting that attempts to replicate the differences in response times have failed (this was the result with the Oxford Center for Neuroethics study as well).
Here is an abstract:
An important quote from the study:
Where to find the study (subscription only):
Kahane, G., K. Wiech, N. Shackel, M. Farias, J. Savulescu and I. Tracey, ‘The Neural Basis of Intuitive and Counterintuitive Moral Judgement’, forthcoming in Social, Cognitive and Affective Neuroscience.
Link on Guy Kahane’s website: http://www.philosophy.ox.ac.uk/members/research_staff/guy_kahane