The constraints of decision theory are only there to help you win; they don’t specify what constitutes a win. Rationality in and of itself cannot constrain what you want, except insofar as what you thought you wanted failed to reflect what you actually wanted (or was just plain inconsistent). Hence the saying: the utility function is not up for grabs.
From the old discussion page:
Talk:The utility function is not up for grabs
Another page that doesn’t link to any Less Wrong blog posts, or any other external references. This automatically qualifies it as either a stub, or as original research, both of which are bad, according to this wiki’s (unwritten?) standards. --PeerInfinity 02:24, 22 October 2009 (UTC)
Added the blogposts link, this is another concept in the something-to-protect/rationalists-should-win &c. cluster of ideas, but I think it deserves its own page. --Zack M. Davis 02:42, 22 October 2009 (UTC)
It seems to me that it should be “the utility function is up for grabs”. Posterior probabilities follow inextricably from prior probabilities; you have no choice in the matter, but you can have your own utility function. --DanielLC 02:25, 14 January 2012 (UTC)
The summary as given is wrong, the point is that utility function is like prior, it’s not something you can change, it’s part of the problem statement. --Vladimir Nesov 15:46, 15 January 2012 (UTC)