Deciding what to think about; is it worthwhile to have universal utility function?
The hunter-gatherer example in the Is risk aversion really irrational? got me thinking about the real world issues with ‘maximizing utility’ and any other simple rule approach to decision making.
The elephant in the room is that universal, effective utility of anything could be very expensive to calculate if you employ any foresight (consider thinking several moves ahead). And once you start estimating utility in different ways depending to the domain, the agent’s behaviour stops being consistent with plain utility maximization. At same time, the solution space of the problems is often very big, meaning that you have immense number of potential choices and you need to perform a lot of utility estimations really quickly to pick the best solution. Think of Chess or Go. The computing time could be better spent elsewhere.
The hunter-gatherer in example can think about the traps and other hunting tools and invent a new one, instead of trying to figure out probability theory or something of this kind.
Inventing a new trap is a case where the number of potential decisions is extremely huge.
I faintly recall a fiction story I’ve read where a smart boy becomes tribe leader—by inventing a better bear trap, not so much by being utterly rational at correctly processing small differences in expected utility when it comes to bets.
He can also think more about berries and look if there’s evidence that other mammals are eating those berries; some plant that is not poisonous to other mammal is very unlikely to hurt a human; some plant that is not eaten by other mammals is very likely to be poisonous to humans as well. He can even feed the berries to some mammal he’d keep alive (i’d imagine keeping animals alive was a fairly straightforward approach to meat preservation).
At the same time, even if that hunter gatherer knew enough math to try to formally calculate his odds, the probabilities are unknown. Indeed there are probability distributions for different degrees of getting sick of berries or not (and different symptoms of sickness), et cetera. We today are just beginning to think how to improve his odds using formal mathematics, and we’re still not sure how to accomplish that, and it is clear that it is going to be very computationally intensive.
As a singular example, I can easily come up with good solutions for that hunter-gatherer by looking into the big solution space that he would have (he’s living in the real world), but it is much harder and much more tedious for me to calculate his odds even in a very simplified example where probabilities of getting sick or winning a duel are exact, and the ‘sick or not sick’ is a binary outcome. That’s with me having a computer at my fingertips, and knowledge of mathematics tens thousands years down the road from hunter gatherer!
Bottom line, it would be very suboptimal for the intelligent hunter gatherer to try to use his intelligence in this particular expected-utility-calculating way to slightly optimize his behaviour (keep in mind that he has no way of estimating probabilities), when lesser amount of good thought would allow him to invent something extremely useful and gain the status.
As a personal success story—I have developed and successfully published a computer game, and made good income on it. The effort that can be spent on decision making—on choosing to implement A or B, is always tightly capped by the other ways of applying effort that would pay off more (implementing both A and B, or searching the solution space more in the hope of coming up with C). It is very rare that putting effort into very careful choice between very few options is the best use of intelligence. It is common in thought experiments but its rare in reality.
I agree with you that, for a normal human, spending too much time/energy doing too precise calculations of outcomes and trying to figure out your utility function is not worth it. Using faster, less precise but less expensive algorithms is often more efficient. My hunter-gatherer would indeed be wiser to find way to be useful to (or, for a darker version of him, manipulate) the tribe, not ponder too much about decision theory.
But that is a different issue than having an universal utility function or not. There an be an universal utility function of a specific human (a mathematical tool to score possible outcomes and select the most desirable ones), while the human don’t know it and don’t actually use it.
But if you want to build a FAI, it then becomes important to figure out human’s utility function.
And if you build an AI (even a limited one, like an AI in a game) it may be interesting to know about decision theory, to know if the AI should have an utility function, if it should be risk averse, …
Indeed. However...
Say, you tell me rules of Chess, and tell me to write a chess engine for a computer chess tournament—software that’s playing chess, run on same hardware. The chess got 3 utility values, really: win>tie>loss .
What will I do?
I will start making functions to evaluate the board position—the simplest one might sum values of pieces—which work a lot like utility functions, but I am going to deviate from maximization of this “utility” whenever I see fit, for this utility doesn’t matter. I will be inventing easy to compute utility function(s) to use to get my agent to the victory. I’d do the same for myself to be able to play it. I’ll have to be maximizing fake utilities, and violating their maximization from time to time.
If I am very advanced, and I make the AI that would be told rules of chess and then play chess (without having been programmed to play chess), such AI will have to invent such substitute functions, for it can not evaluate true utility of any move that’s far from the loss/win/tie, and it will lose almost all of the pieces before it’s choices being being driven by it’s foresight of it’s demise. This will be the case even if the AI is to run on strongly superhuman hardware that does 10^30 FLOPS (think Dyson Spheres). It will still get it merry ass handed to it even by deep blue (or Kasparov), if it won’t meta-strategize and invent utility functions that lead to victory.