preferences:decision theory :: data:code
I’d like to present a couple thoughts. While I am somewhat confident in my reasonning, my conclusions strongly contradict what I perceive (possibly incorrectly) to be the concensus around decision theory on LessWrong. This consensus has been formed by people who have spent more time than me thinking about it, and are more intelligent than I am. I am aware of that, this is strong evidence that I am mistaken or obvious. I believe nonetheless the argument I’m about to make is valuable and should be heard.
It is argued that the key difference between Newcomb’s problem and Solomon’s problem is that precommitment is useful in the former and useless in the latter. I agree that the problems are indeed different, but I do not think that is the fundamental reason. The devil is in the details.
Solomon’s problem states that
- There is a gene that causes people to chew gum and to develop throat cancer
—Chewing gum benefits everyone
It is generally claimed that EDT would decide not to chew gum, because doing so would place the agent in a state where its expected utility is reduced. This seems incorrect to me. The ambiguity is in what is meant by “causes people to chew gum”. If the gene really causes people to chew gum, then that gene by definition affects that agent’s decision theory, and the hypothesis that it is also following EDT is contradictory. What is generally meant is that having this gene induces a preference to chew gum, which is generally acted upon by whatever decision algorithm is used. An EDT agent must be fully aware of its own preferences, otherwise it could not calculate its own utility, therefore, the expected utility of chewing gum must be calculated conditional on having a preexisting or non preexisting taste for gum. In a nutshell, an EDT agent updates not on his action to chew gum, but on his desire to do so.
I’ve established here a distinction between preferences and decision theory. In fact, the two are interchangeable. It is always possible to hard code preferences in the decision theory, and vice versa. The distinction is very similar to the one drawn between code and data. It is an arbitrary but useful distinction. Intuitively, I believe hard coding preferences in the decision algorithm is poor design, though I do not have a clear argument why that is.
If we insist on preferences being part of the decision algorithm, the best decision algorithm for solomon’s problem is the one that doesn’t have a cancer causing gene. If the algorithm is EDT, then liking gum is a preference, and EDT makes the same decision as CDT.
Let’s now look at Newcomb’s problem. Omega’s decision is clearly not based on a subjective preference for one box or two box (let’s say an aesthetic preference for example). Omega’s decision is based on our decision algorithm itself. This is the key difference between the two problems, and this is why precommitment works for Newcomb’s and not Solomon’s.
Solomon’s problem is equivalent to this problem, which is not Newcomb’s
- If Omega thinks you were born loving Beige, he puts $1,000 in box Beige and nothing in box Aquamarine.
- Otherwise, he puts $1,000 in box Beige and nothing in box Aquamarine.
In this problem, both CDT and EDT (correctly) two box. Again, this is because EDT knows that it loves beige.
Now the real Newcomb’s problem. I argue that an EDT agent should integrate his own decision as evidence.
- If EDT’s decision is to two-box, then Omega’s prediction is that EDT two boxes and EDT should indeed two-box.
- If EDT’s decision is to one-box, then Omega’s prediction is that EDT one box, and EDT should two-box.
Since EDT reflects on his own decision, it can only be the only fixed point which is to two box.
Both CDT and EDT decide to chew gum and to two box.
If we’re out shopping for decision algorithms (TDT, UDT...), we might as well shop for a set of preferences, since they can be interchangeable. It is clear that some preferences allow winning, when variable sum games are involved. This has been implemented by evolution as moral preferences, not as decision algorithms. One useful preference is the preference to keep one’s word. Such a preference allows to pay Parfit’s hitchiker without involving any preference reversal. Once you’re safe, you do not try not to pay, because you genuinely prefer not breaking your promise than keeping the money. Yes, you could have preferences to two box, but there is no reason why you should catter in advance to crazy cosmic entities rewarding certain algorithms or preferences. Omega is no more likely than the TDT and UDT minimizer, evil entities known for torturing TDT and UDT practionners.
Edit: meant to write EDT two-boxes, which is the only fixed point.
In the future, I think that type of post should go in the Discussion section.
Here’s an intuition pump:
10% of the population sunburns easily. 90% of people who sunburn easily wear sunscreen. 10% of the people who don’t sunburn easily wear sunscreen. Regardless of whether or not you sunburn easily, wearing sunscreen will reduce the chances of getting sunburn. You don’t know if you sunburn easily or not.
EDT looks at this data and says, “It’s better to be a randomly chosen member of the group of people who don’t wear sunscreen than a randomly chosen member of the group of people who do. Therefore, don’t wear sunscreen.” And that seems like the wrong answer, because your decision whether or not to wear sunscreen doesn’t actually affect whether or not you sunburn easily. In other words, the problem with EDT is that it can’t handle “Simpson’s Paradox”.
According to wikipedia, the definition of EDT is
This is not the same as “being a randomly chosen member of a group of people...” and I’ve explained why. The information about group membership is contained in the filtration.
Moved to Discussion.
This is actually not what is meant when considering Solomon’s problem. They really do mean the actual decision.
This case is handled in the previous sentence. If this is your actual decision, and your actual decision is the product of a decision algorithm, then your decision algorithm is not EDT.
To put it another way, is your decision to chew gum determined by EDT our by your genes? Pick one.
It can be both. Causation is not exclusionary. I’m suggesting that you are mistaken about the aforementioned handling.
No it can’t. If you use a given decision theory, your actions are entirely determined by your preferences and your sensory inputs.
wedrifid might well be making the point that your genes determine your choice, via your decision theory. i.e. Your genes give you EDT, and then EDT makes you not chew gum. I’m not sure how that affects the argument though.
The claim is generally that EDT chooses not to chew gum.
Thanks, fixed.
You’re saying EDT causes you not to chew gum because cancer gives you EDT? Where does the gum appear in the equation?
Yes, but this dependence factors through the strategy that the algorithm produces. Read Eliezer’s TDT document(pdf) where he talks about ‘action-determined’ and ‘decision-determined’ problems. Whereas CDT only ‘wins’ at the former, TDT ‘wins’ at both. Note that in a decision-determined problem, Omega is allowed but a TDT-minimizer is not.
You appear to mean: When an EDT agent hypothesizes “suppose my decision were X” it’s subsequently allowed to say “so in this hypothetical I’ll actually do Y instead.”
But that’s not how EDT works—your modification amounts to a totally different algorithm, which you’ve conveniently named “EDT”.
...then Omega’s prediction is that EDT will two-box and oops—goodbye prize.
EDT measures expected value after the action has been taken, but the output of EDT has no reason to be ignored by EDT if it is relevant to the calculation.
It loses, but it is generally claimed that EDT one boxes.
The problem generalises to any situation requiring cooperation based on mutual knowledge of the other agent’s decision process when there is not the option to introduce new constraints in the environment.
Yes, the causality is from the decision process to the reward. The decision process may or may not be known to the agent, but its preferences are (data can be read, but the code can only be read if introspection is available).
You can and should self-modify to prefer acting in such a way that you would benefit from others predicting you would act a certain way. You get one-boxing behavior in Newcomb’s and this is still CDT/EDT (which are really equivalent, as shown).
Yes, you could implement this behavior in the decision algorithm itself, and yes this is very much isomorphic. Evolution’s way to implement better cooperation has been to implement moral preferences though, it feels like a more natural design.
I suggest that what was ‘shown’ was that you do not understand the difference between CDT and EDT.
That’s certainly possible, it’s also possible that you do not understand the argument.
To make things absolutely clear, I’m relying on the following definition of EDT
Policy that picks action a = argmax( Sum( P( Wj | W, ai ). U( Wj ), j ) , i ) Where {ai} are the possible actions, W is the state of the world, P( W’ | W, a ) the probability of moving to state of the world W’ after doing a, and U is the utility function.
I believe the argument I made in the case of Solomon’s problem is the clearest and strongest, would you care to rebut it?
I’ve challenged you to clarify through which mechanism someone with a cancer gene would decide to chew gum, and you haven’t answered this properly.
If your decision algorithm is EDT, the only free variables that will determine what your decisions are are going to be your preferences and sensory input.
The only way the gene can cause you to chew gum in any meaningful sense is to make you prefer to chew gum.
An EDT agent has knowledge of its own preferences. Therefore, an EDT agent already knows if it falls in the “likely to get cancer” population.
The combination:
Uncontraversial understanding by academic orthodoxy
General position by those on lesswrong
My parsing of your post
Observation of your attempts to back up your argument when it was not found to be persuasive by myself or others
… is sufficient to give rather high confidence levels. It really is a huge claim you are making, to dismiss the understanding of basically the rest of the world regarding how CDT and EDT apply to the trivial toy problems that were designed to test them.
There is altogether too much deduction of causal mechanisms involved in your “EDT” reasoning. And the deductions involved rely on a premise (the second dot point) that just isn’t a part of either the problem or ‘genes’.
I’m making a simple, logical argument. If it’s wrong, it should be trivial to debunk. You’re relying on an outside view to judge; it is pretty weak.
As I’ve clearly said, I’m entirely aware that I’m making a rather controversial claim. I never bother to post on lesswrong, so I’m clearly not whoring for attention or anything like that. Look at it this way, in order to present my point despite it being so unorthodox, I have to be pretty damn sure it’s solid.
The second dot point is part of the problem description. You’re saying it’s irrelevant, but you can’t just parachute a payoff matrix where causality goes backward in time.
Find any example you like, as long as they’re physically possible, you’ll either have the payoff tied to your decision algorithm (Newcomb’s) or to your preference set (Solomon’s).
Did you intend for Omega to act differently in these two situations?
I agree that EDT should one box, but the your previous two lines suggest the fixed point is at two-boxing.
Typo, I do mean that EDT two boxes.