Seems like a simple and reasonable answer to this problem is that I would take the box with the million dollars, rather than the box with the thousand dollars and the empty box. It seems the main question is, “But why?”. So here is my reasoning: Omega has shown over 99% accuracy in providing results dependent on people’s choices. Box B has 100,000% better rewards than Box A, such that if there is even 0.1% chance that taking Box A will lose those rewards, it is irrational to also take Box A. As I have seen no evidence that Omega has left, it is not even certain that my choice of actions now will have not effect on the contents of the opaque box (only a fool would be certain that just because he “saw Omega fly away” that said superintelligence is not hiding nor has left behind an observing agent). As each of these choices would cause my choice to be only Box B, it is almost certain that Omega has seen likewise and put the $1,000,000 in Box B.
I suspect the problem people seem to have with this is because they think they are outside of the game. But the game description itself says that (a very accurate model of) you is in the game, and that therefore your (modeled) choices, including second doubts and your desire not to leave that last $1,000, will (if modeled correctly) affect the contents of Box B. No, Omega is not rewarding irrationality. Omega is giving a large reward to those who trust in Omega’s judgement, and a smaller reward to those who arrogantly think they can cheat the game he set up.
Incidentally, it is currently possible to achieve total happiness, or perhaps a close approximation. A carefully implanted electrode to the right part of the brain, will be more desirable than food to a starving rat, for example. While this part of the brain is called the “pleasure center”, it might rather be about desire and reward instead. Nevertheless, pleasure and happiness are by necessity mental states, and it should be possible to artificially create these.
Why should a man who is perfectly content, bother to get up to eat, or perhaps achieve something? He may starve to death, but would be happy to do so. And such a man will be content with his current state, which of course is contentment, and not at all resent his current state. Even a less invasive case, where a man is given almost everything he wants, yet not so much so that he does not eventually become dissatisfied with the amount of food in his belly and decide to put more in, even so there will be higher level motivations this man will lose.
While I consider myself a utilitarian, and believe the best choices are those that maximize the values of everyone, I cannot agree with the above situation. For now, this is no problem because people in their current state would not choose to artificially fulfill their desires via electrode implants, nor is it yet possible to actually fulfill everyone’s desires in the real world. I shall now go and rethink why I choose a certain path, if I cannot abide reaching the destination.