Three points in response to Eliezer’s post and one of his replies:
* A limited time horizon works better than he says. If an AI wants to put its world into a state desired by humans, and it knows that the humans don’t want to live in a galaxy that will be explode in a year, then an AI that closes its books in 1000 years will make sure that the galaxy won’t explode one year later.
* An unbounded utility works worse than he says. Recall the ^^^^ operator originally by Knuth (see http://en.wikipedia.org/wiki/Knuth%27s_up-arrow_notation) that was used in the Pascal’s Mugging article at http://lesswrong.com/lw/kd/pascals_mugging_tiny_probabilities_of_vast/.
If one allows unbounded utilities, then one has allowed a utility of about 3^^^^3 that has no low-entropy representation. In other words, there isn’t enough matter to represent a utility.
Humans have heads of a limited size that don’t use higher math to represent their desires, so bounding the utility function doesn’t limit our ability to describe human desire.
* Ad-hominem is a fallacy. The merit of a proposed FAI solution is a function of the solution, not who proposed it or how long it took them. An essential step toward overcoming bias is to train oneself not to commit well-known fallacies. There’s a good list in “The Art of Controversy” by Schopenhauer, see http://www.gutenberg.org/etext/10731.
Of course, I’m bothering to say this because I have a proposed solution out. See http://www.fungible.com/respect/paper.html.
Hugo Mercier’s citation above for “Believe it or Not” by Hasson et al. wants money to give you the article. The article is available for free from Hasson’s home page at:
http://home.uchicago.edu/~uhasson/
The direct URL is:
http://home.uchicago.edu/~uhasson/Belief.pdf