I feel again as if I do not understand what Timeless Decision Theory or Updateless Decision Theory is (or what it’s for; what it adds to ordinary decision theory). Can anyone help me? For example, by providing the simplest possible example of one of these “decision theories” in action?
Suppose we have an agent that cares about something extremely simple, like number of paperclips in the world. More paperclips is a better world. Can someone provide an example of how TDT or UDT would matter, or would make a difference, or would be applied, by an entity which made its decisions using that criterion?
Naive decision theory: “Choose the action that will cause the highest expected utility, given what I know now.”
Timeless decision theory: “Choose the action that I wish I had precommitted to, given what I know now.”
Updateless decision theory: “Choose a set of rules that will cause the highest expected utility given my priors, then stick to it no matter what happens.”
There’s nothing that prevents a UDT agent from behaving as if it were updating; that’s what I surmise would happen in more normal situations where Omega isn’t involved. But if ignoring information is the winning move, TDT can’t do that.
TDT and UDT are intended to solve Newcomb’s problem and the prisoner’s dilemma and those are surely the simplest examples of their strengths. It is fairly widely believed that, say, causal decision theory two-boxes and defects, but I would rather say that CDT simply doesn’t understand the statements of the problems. Either way, one-boxing and arranging mutual cooperation are improvements.
Here, so far as I can understand it, is UDT vs. ordinary DT for paper clips:
Ordinary DT (“ODT”) says: at all times t, act so as to maximize the number of paper clips that will be observed at time (t + 1), where “1″ is a long time and we don’t have to worry about discount rates.
UDT says: in each situation s, take the action that is returns the highest value on an internal lookup table that has been incorporated into me as part of my programming, which, incidentally, was programmed by people who loved paper clips.
Suppose ODT and UDT are fairly dumb, say, as smart as a cocker spaniel.
Suppose we put both agents on the set of the movie Office Space. ODT will scan the area, evaluate the situation, and simulate several different courses of action, one of which is bending staples into paper clips. Other models might include hiding, talking to accountants, and attempting to program a paper clip screensaver using Microsoft Office. The model that involves bending staples shows the highest number of paper clips in the future compared to other models, so the ODT will start bending staples. If the ODT is later surprised to discover that the boss has walked in and confiscated the staples, it will be “sad” because it did not get as much paper-clip utility as it expected to, and it will mentally adjust the utility of the “bend staples” model downward, especially when it detects boss-like objects. In the future, this may lead ODT to adopt different courses of behavior, such as “bend staples until you see boss, then hide.” The reason for changing course and adopting these other behaviors is that they would have relatively higher utility in its modeling scheme.
UDT will scan the area, evaluate the situation, and categorize the situation as situation #7, which roughly corresponds to “metal available, no obvious threats, no other obvious resources,” and lookup the correct action for situation #7, which its programmers have specified is “bend staples into paper clips.” Accordingly, UDT will bend staples. If UDT is later surprised to discover that the boss has wandered in and confiscated the staples, it will not care. The UDT will continue to be confident that it did the “right” thing by following its instructions for the given situation, and would behave exactly the same way if it encountered a similar situation.
UDT sounds stupider, and, at cocker-spaniel levels of intelligence, it undoubtedly is. That’s why evolution designed cocker-spaniels to run on ODT, which is much more Pavlovian. However, UDT has the neat little virtue that it is immune to a Counterfactual Mugging. If we could somehow design a UDT that was arbitrarily intelligent, it would both achieve great results and win in a situation where ODT failed.
Here, so far as I can understand it, is Tyrell’s UDT vs. ordinary DT for paper clips:
For god’s sake, don’t call it my UDT :D. My post already seems to be giving some people the impression that I was suggesting some amendment or improvement to Wei Dai’s UDT.
If it’s any consolation, the last bit of understanding of the original Wei Dai’s post (the role of execution histories, prerequisite to being able to make this correction) dawned on me only last week, as a result of a long effort for developing a decision theory of my own that only in retrospect turned out to be along roughly the same lines as UDT.
A convergence like that makes both UDT and your decision theory more interesting to me. Is the process of your decision theory’s genesis detailed on your personal blog? In retrospect, was your starting place and development process influenced heavily enough by LW/OB/Wei Dai to screen out the coincidence?
I call it “ambient control”. This can work as an abstract:
You, as an agent, determine what you do, and so have the power to
choose which statements about you are true. By making some statements
true and not others, you influence the truth of other statements that
logically depend on the statements about you. Thus, if you have
preference about what should be true about the world, you can make
some of those things true by choosing what to do. Theories of
consequences (partially) investigate what becomes true if you make a
particular decision. (Of course, you can’t change what’s true, but you
do determine what’s true, because some truths are about you.)
Longer description here. I’ll likely post on some aspects of it in the future, as the idea gets further developed. There is a lot of trouble with logical strength of theories of consequences, for example. There is also some hope to unify logical and observational uncertainty here, at the same time making the decision algorithm computationally feasible (it’s not part of the description linked above).
I feel again as if I do not understand what Timeless Decision Theory or Updateless Decision Theory is (or what it’s for; what it adds to ordinary decision theory). Can anyone help me? For example, by providing the simplest possible example of one of these “decision theories” in action?
Suppose we have an agent that cares about something extremely simple, like number of paperclips in the world. More paperclips is a better world. Can someone provide an example of how TDT or UDT would matter, or would make a difference, or would be applied, by an entity which made its decisions using that criterion?
This is my vague understanding.
Naive decision theory: “Choose the action that will cause the highest expected utility, given what I know now.”
Timeless decision theory: “Choose the action that I wish I had precommitted to, given what I know now.”
Updateless decision theory: “Choose a set of rules that will cause the highest expected utility given my priors, then stick to it no matter what happens.”
If this is accurate, then I don’t see how UDT can generally be better than TDT.
UDT would be better in circumstances where you suspect that your ability to update accurately is compromised.
I’m assuming that the priors for UDT were set at some past time.
UDT gives the money in the counterfactual mugging thought experiment, TDT doesn’t.
There’s nothing that prevents a UDT agent from behaving as if it were updating; that’s what I surmise would happen in more normal situations where Omega isn’t involved. But if ignoring information is the winning move, TDT can’t do that.
TDT and UDT are intended to solve Newcomb’s problem and the prisoner’s dilemma and those are surely the simplest examples of their strengths. It is fairly widely believed that, say, causal decision theory two-boxes and defects, but I would rather say that CDT simply doesn’t understand the statements of the problems. Either way, one-boxing and arranging mutual cooperation are improvements.
Here, so far as I can understand it, is UDT vs. ordinary DT for paper clips:
Ordinary DT (“ODT”) says: at all times t, act so as to maximize the number of paper clips that will be observed at time (t + 1), where “1″ is a long time and we don’t have to worry about discount rates.
UDT says: in each situation s, take the action that is returns the highest value on an internal lookup table that has been incorporated into me as part of my programming, which, incidentally, was programmed by people who loved paper clips.
Suppose ODT and UDT are fairly dumb, say, as smart as a cocker spaniel.
Suppose we put both agents on the set of the movie Office Space. ODT will scan the area, evaluate the situation, and simulate several different courses of action, one of which is bending staples into paper clips. Other models might include hiding, talking to accountants, and attempting to program a paper clip screensaver using Microsoft Office. The model that involves bending staples shows the highest number of paper clips in the future compared to other models, so the ODT will start bending staples. If the ODT is later surprised to discover that the boss has walked in and confiscated the staples, it will be “sad” because it did not get as much paper-clip utility as it expected to, and it will mentally adjust the utility of the “bend staples” model downward, especially when it detects boss-like objects. In the future, this may lead ODT to adopt different courses of behavior, such as “bend staples until you see boss, then hide.” The reason for changing course and adopting these other behaviors is that they would have relatively higher utility in its modeling scheme.
UDT will scan the area, evaluate the situation, and categorize the situation as situation #7, which roughly corresponds to “metal available, no obvious threats, no other obvious resources,” and lookup the correct action for situation #7, which its programmers have specified is “bend staples into paper clips.” Accordingly, UDT will bend staples. If UDT is later surprised to discover that the boss has wandered in and confiscated the staples, it will not care. The UDT will continue to be confident that it did the “right” thing by following its instructions for the given situation, and would behave exactly the same way if it encountered a similar situation.
UDT sounds stupider, and, at cocker-spaniel levels of intelligence, it undoubtedly is. That’s why evolution designed cocker-spaniels to run on ODT, which is much more Pavlovian. However, UDT has the neat little virtue that it is immune to a Counterfactual Mugging. If we could somehow design a UDT that was arbitrarily intelligent, it would both achieve great results and win in a situation where ODT failed.
For god’s sake, don’t call it my UDT :D. My post already seems to be giving some people the impression that I was suggesting some amendment or improvement to Wei Dai’s UDT.
Edited. [grin]
If it’s any consolation, the last bit of understanding of the original Wei Dai’s post (the role of execution histories, prerequisite to being able to make this correction) dawned on me only last week, as a result of a long effort for developing a decision theory of my own that only in retrospect turned out to be along roughly the same lines as UDT.
A convergence like that makes both UDT and your decision theory more interesting to me. Is the process of your decision theory’s genesis detailed on your personal blog? In retrospect, was your starting place and development process influenced heavily enough by LW/OB/Wei Dai to screen out the coincidence?
I call it “ambient control”. This can work as an abstract:
Longer description here. I’ll likely post on some aspects of it in the future, as the idea gets further developed. There is a lot of trouble with logical strength of theories of consequences, for example. There is also some hope to unify logical and observational uncertainty here, at the same time making the decision algorithm computationally feasible (it’s not part of the description linked above).