This post aims to provide a strong philosophical foundation for logical counterfactuals, while sketching out an informal scheme that will hopefully be formalised further in the future. I believe that providing such a philosophical foundation is important for the same reasons that Sam listed in Motivating a Semantics of Logical Counterfactuals.
Introductory Material
Miri’s Functional Decision Theory (FDT) paper defines subjunctive dependence to refer to the situation where two decision process involve the same calculation. This can result from causation, such as if I calculate an equation, then tell you the answer and we both implement a decision based on this sum. It can also occur non-causally, such as a prediction being linked to a decision as per Newcomb’s problem. The problem of logical counterfactuals can be characterised as figuring out what processes subjunctively depend on other processes, so that we can apply FDT.
In the Co-operation Game, I argued that logical counterfactuals are more about your knowledge of the state of the world than the world itself. Suppose there are two people who can choose A or B. Suppose that a predictor knows that both people will choose A conditional on them being told one of the following two facts a) the other person will choose A b) the other person will choose the same as you. Then whether your decision is modelled to subjunctively depend on the other person depends on which of the two facts you are told. Going further than the original post, one might be told a) and the other b), so that the first sees themselves as not subjunctively linked to the second, while the second sees themselves as subjunctively linked to the first.
“Your knowledge of the state of the world” can be explicated as being about internally consistent counterfactuals, which we’ll label Raw Counterfactuals. When there are multiple raw counterfactuals consistent with your state of knowledge, you can pick the one with the highest utility.
However, there will also be cases where only a single counterfactual is consistent with your state of knowledge, which results in a rather trivial problem. Consider for example Transparent Newcomb’s Problem, where a perfect predictor places the million in a transparent box if and only if it predicts that you will one-box if it does. If you see the million, you know that you must have one-boxed so it doesn’t strictly make sense to ask what you should do in this situation. Instead, we’ll have to ask something slightly different instead. So, I’ve slightly modified my position since writing the Co-operation Game: in some situations logical counterfactuals will be defined relative to an imagined, rather than actual epistemic state. We will construct these states by erasing some information as described later in the post.
Other degenerative cases include when you already know what decision you’ll make or when you have the ability to figure it out. For example, when you have perfect knowledge of the environment and the agent, unless you run into issues with unprovability. Note that degeneracy is more common than you might think since knowing, for example, that it is a utility maximiser, tells you its exact behaviour in situations without options that are tied. Again, in these cases, the answer to the question, “What should the agent do?” is, “The only action consistent with the problem statement”. However, as we’ll see, it is possible sometimes to make these questions less trivial if you’re willing to accept a slight tweak to the problem statement.
We’ll consider two kinds of problems, acknowledging that these aren’t the only types. In external problems, we imagine decisions from the perspective of a theoretical, unbounded, non-embedded observer who exists outside of the problem statement. Clearly we can’t fully adopt the perspective of such an external agent, but describing the high-level details will usually suffice. Critically, in the external perspective, the observer can have goals, such as choosing the agent with the maximum utility, without those being the goals of the agent within the problem.
In (fully) reflective problems, we imagine decisions from the perspective of an agent considering its own decisions or potential decisions with full knowledge of its own source code. These problems will complicate the counterfactuals since the agent’s goals limit the kind of agent that it could be. For example, an agent that wants to maximise utility should only search over the possibility space where it is a utility maximiser.
Making this distinction more explicit: An external problem would ask “What decision maximises utility?”, as opposed to a reflective problem which asks: “What decision maximises utility for a utility maximiser?”. This distinction will mainly be important here in terms of when it makes a problem trivial or not.
The external/reflective distinction is very similar to the difference between embedded and non-embedded problems, but external problems can include embedded agents, just from the perspective of a non-embedded agent. So we can do a surprising amount of our theorising from within the external perspective.
Raw Counterfactuals
Raw counterfactuals are produced as follows: Starting with the territorywe use some process to produce a causal model. We can then imagine constructing different models by switching out or altering parts of the model. These represent a coherent concept seperate from any discussion of decisions. In so far as we care about what could have been, we need to ultimately relate our claims to raw counterfactuals as inconsistent models could not have been.
Causal Decision Theory uses its own notion of counterfactuals, which we’ll term Decision Counterfactuals. These are created by performing world surgery on models of causal processes. Unlike raw counterfactuals, decision counterfactuals are inconsistent. Suppose that you actually defect in the Prisoner’s Dilemma, but we are considering the counterfactual where you cooperate. Up until the point of the decision you are the kind of person who defects, but when we arrive at the decision, you magically cooperate.
Decision counterfactuals are useful because they approximate raw counterfactuals. Performing world surgery all the way back in time requires would require a lot of work. In theoretical decision problems it is usually easy to imagine a raw counterfactual that would match the problem description and provide the same answer as the decision counterfactual. In practical decision problems, we don’t have the data to do this.
Unfortunately, this approximation breaks down when performing world surgery to make a counterfactual decision consistent with the past requires us to change an element of the environment that is important for the specific problem. For example, in Newcomb’s problem, changing your current decision requires changing your past self which involves changing a predictor which is considered part of the environment. In this case, it makes sense to fall back to raw counterfactuals and build a new decision theory on top.
Functional decision theory, as normally characterised, is closer to this ideal as world surgery isn’t just performed on your decision, but also on all decisions that subjunctively depend on you. This removes a bunch of inconsistencies, however, we’ve still introduced an inconsistency by assuming that f(x)=b when f(x) really equals a. The raw counterfactual approach provides a stronger foundation because it avoids this issue. However, since proof-based FDT is very effective at handling reflective problems, it would be worthwhile rebuilding it upon this new foundation.
Let’s consider Newcomb’s problem from the external perspective. The external observer is trying to maximise utility rather than the agent within the problem, so there is no restriction on whether the agent can one-box or two-box whilst being consistent with the problem statement. We can then immediately observe that if we use raw counterfactuals, the reward only depends on the agent’s ultimate decision and agents that one-box score better than those who don’t. Simple cases like this which allow multiple consistent counterfactuals don’t require erasure.
On the other hand, there are problems which only allow a single raw counterfactual and hence require us to tweak the problem to make it well-defined. Consider, for examples, Transparent Newcomb’s, where if you see money in the box, you know that you will receive exactly $1 million. Some people say this fails to account for the agent in the simulator, but it’s entirely possible that Omega may be able to figure out what action you will take based on high level reasoning, as opposed having to run a complete simulation of you. We’ll describe later a way of tweaking the problem statement into something that is both consistent and non-trivial.
General Approach to Logical Counterfactuals
We will now attempt to produce a more solid theory of external problems using FDT. This will allow us to interpret decision problems where only one decision is consistent with the problem statement in a non-trivial way.
FDT frames logical counterfactuals as “What would the world be like if f(x)=b instead of a?” which doesn’t strictly make sense as noted in the discussion on raw counterfactuals. Two points: a) I think it should be clear that this question only makes sense in terms of thinking of perturbations of the map and not as a direct claim about the territory (see map and territory). b) We’ll address this problem by proposing a different approach for foundations, which these proof-based approaches should ultimately be justified in terms of.
There are two possible paths towards a more consistent theory of logical counterfactuals for these situations. In both cases we interpret the question of what it would mean to change the output of a function as an informal description of a similar question that is actually well-defined. The first approach is see what consequences can be logically deduced from f(x)=b while implementing a strategy to prevent us from deducing incorrect statements from the inconsistencies. This is often done by playing chicken with the universe. We will term this a paraconsistent approach, even though it doesn’t explicitly make use of paraconsistent logic as it is paraconsistent in spirit.
An alternative approach would be to interpret this sentence as making claims about raw counterfactuals. In FDT terms, the raw counterfactual approach finds an f’ such that f’(x)=b and also with certain as of yet unstated similarities to f and substitute this into all subjunctively linked processes. The paraconsistent approach is easier to do informally, but I suspect that the raw counterfactually approach would be more amenable to formalisation and provides more philosophical insight into what is actually going on. In so far as the paraconsistent approach may be more convenient for an implementation perspective than the first, we can justify it by tying it to raw counterfactuals.
Decisions
Now that we’ve outlined the broad approach, we should dig more into the question of what exactly it means to make a decision. As I explained in a previous post, there’s a sense in which you don’t so much ‘make’ a decision as implement one. If you make something, it implies that it didn’t exist before and now it does. In the case of decisions, it nudges you towards believing that the decision you were going to implement wasn’t set, then you made a decision, and then it was. However, when “you” and the environment are defined down to the atom, you can only implement one decision. It was always the case from the start of time that you were going to implement that decision.
We note that if you have perfect information about the agent and the environment, you need to forget or at least pretend to forget some information about the agent so that we can produce counterfactual versions of the agent who decide slightly differently. See Shminux’s post on Logical Counterfactuals are Low Res for a similar argument, but framed slightly differently. The key difference is that I’m not suggesting just adding noise to the model, but forgetting specific information that doesn’t affect the outcome.
In Transparent Newcomb’s, it would be natural to erase the knowledge that the box is full. This would then result in two counterfactuals: a) the one where the agent sees an empty box and two boxes, b) the one where the agent sees a full box and one boxes. It would be natural to relax the scope of who we care about from the agent who sees a million in the box to the agent at the end of the problem regardless of what they see. If we do so, we then have a proper decision problem and we can see that one-boxing is better.
Actually there’s a slight hitch here. In order to define the outcome an agent receives, we need to define what the predictor will predict when an agent sees the box containing the million. But it is impossible to place a two-boxer in this situation. We can resolve this by defining the predictor as simulating the agent responding to an input representing an inconsistent situation as I’ve described in Counterfactuals for Perfect Predictors.
In order to imagine a consistent world, when we imagine a different “you”, we must also imagine the environment interacting with that different “you” so that, for example, the predictor makes a different prediction. Causal decision theorists construct these counterfactuals incorrectly and hence they believe that they can change their decision without changing the prediction. They fail to realise that they can’t actually “change” their decision as there is a single decision that they will inevitably implement. I suggest replacing “change a decision” with “shift counterfactuals” when it is important to be able to think clearly about these topics. It also clarifies why the prediction can change without backwards causation (my previous post on Newcomb’s problem contains further material on why this isn’t an issue).
Erasure
Here’s how the erasure may proceed for the example of Transparent’s Newcomb Problem. Suppose we erase all information about what decision the agent is going to make. This also requires erasing the fact that you see a million in the transparent box. Then we look at all counterfactually possible agents and notice that the reward depends only on whether you are an agent who ultimately one-boxes or an agent who two-boxes. Those who one-box see the million and then receive it, those who two-box see no money in the transparent box and receive $1000 only. The counterfactual involving one-boxing performs better than that involving two-boxing, so we endorse one-boxing.
Things become more complicated if we want to erase less information about the agent. For example, we might want an agent to know that it is a utility maximiser as this might be relevant to evaluating the outcome the agent will receive from future decisions. Suppose that if you one-box in Transparent Newcomb’s you’ll then be offered a choice of $1000 or $2000, but if you two-box you’ll be offered $0 or $10 million. We can’t naively erase all information about your decision process in order to compute a counterfactual of whether you should one-box or two-box. Otherwise, we end up with a situation where, for example, there are agents who one-box and get different rewards. Here the easiest solution is to “collapse” all of the decisions and ask about a policy instead that covers all three decisions that may be faced. Then we can calculate the expectations without producing any inconsistencies in the counterfactuals as it then becomes safe to erase the knowledge that it is a utility maximiser.
The 5 and 10 problem doesn’t occur with the forgetting approach. Compare: If you keep the belief that you are a utility maximiser then the only choice you can implement is 10 so we don’t have a decision problem. We can define all the possible strategies as follows, p probability of choosing 5 and 1-p probability of choosing 10, so forget everything about yourself except that you are one of these strategies. There’s no downside as there is no need for an agent to know whether or not it is a utility maximiser. So we can solve the 5 and 10 problem without epsilon exploration.
When will we be able to use this forgetting technique? One initially thought might be the same scope FDT is designed to be optimal on—problems where the reward only depends on your outputs or predicted outputs. Because only the output or predicted output matters and not the algorithm, these can be considered fair, unlike a problem where an alphabetical decision theorist (picks the first decision ordered in alphabetical order) is rewarded and every other type of agent is punished.
However, some problems where this condition doesn’t hold also seem fair. Like suppose there are long programs and short programs (in terms of running time). Further suppose programs can output either A or B. The reward is then determined purely based on these two factors. Now suppose that there exists a program that can calculate the utilities in each of these four worlds and then based upon this either terminate immediately or run for a long time and then output its choice of A or B. Assume that if it terminates immediately after the decision it’ll qualify as a short program, while if it terminates after a long time it is a long program. Then, as a first approximation, we can say that this is also a fair problem since it is possible to win in all cases.
It’s actually slightly more complex than this. An AI usually doesn’t have to win on only one problem. Adding code to handle more situations will extend the running time and may prevent such an AI from always being able to choose the dominant option. An AI might also want to do things like double check calculations or consider whether it is actually running on consistent outcomes, so winning the problem might put limitations on the AI in other ways.
But nonetheless, so long as we can write such a program that picks the best option, we can call the problem “fair” in a limited sense. It’s possible to extend our definition of “fair” further. Like suppose that it’s impossible to analyse all the options and still return in a short amount of time. This isn’t a problem if the maximum utility is in a long option.
In regards to running time, we can also end up with a non-binary notion of “fair” according to how much extra processing a program can squeeze in before having to decide the short option. This limits the ability of the AI to check/recheck its work and examine its philosophical assumptions before having to make the decision.
Final Thoughts
Logical counterfactuals are often framed in such a way that it seems that we should be building a model of subjunctive dependence directly from the atoms in the universe. Instead we produce these from causal model that identifies the current agent and a model of forgetting. This makes our job much easier as it allows us to disentangle this problem from many of the fundamental issues in ontology and philosophy of science.
Comparing the method in this post to playing chicken with the universe: using raw counterfactuals clarifies that such proof-based methods are simply tricks that allow us to act as though we’ve forgotten without forgetting.
In general, I would suggest that logical counterfactuals are about knowing which information to erase such that you can produce perfectly consistent counterfactuals. Further, I would suggest that if you can’t find information to erase that produces perfectly consistent counterfactuals then you don’t have a decision theory problem. Future work could explore exactly when this is possible and general techniques for making this work as the current exploration has been mainly informal.
Notes:
I changed my terminology from Point Counterfactuals to Decision Counterfactuals and Timeless Counterfactuals to Raw Counterfactuals in this post as this better highlights where these models come from.
This post was written with the support of the EA Hotel
Deconfusing Logical Counterfactuals
This post aims to provide a strong philosophical foundation for logical counterfactuals, while sketching out an informal scheme that will hopefully be formalised further in the future. I believe that providing such a philosophical foundation is important for the same reasons that Sam listed in Motivating a Semantics of Logical Counterfactuals.
Introductory Material
Miri’s Functional Decision Theory (FDT) paper defines subjunctive dependence to refer to the situation where two decision process involve the same calculation. This can result from causation, such as if I calculate an equation, then tell you the answer and we both implement a decision based on this sum. It can also occur non-causally, such as a prediction being linked to a decision as per Newcomb’s problem. The problem of logical counterfactuals can be characterised as figuring out what processes subjunctively depend on other processes, so that we can apply FDT.
In the Co-operation Game, I argued that logical counterfactuals are more about your knowledge of the state of the world than the world itself. Suppose there are two people who can choose A or B. Suppose that a predictor knows that both people will choose A conditional on them being told one of the following two facts a) the other person will choose A b) the other person will choose the same as you. Then whether your decision is modelled to subjunctively depend on the other person depends on which of the two facts you are told. Going further than the original post, one might be told a) and the other b), so that the first sees themselves as not subjunctively linked to the second, while the second sees themselves as subjunctively linked to the first.
“Your knowledge of the state of the world” can be explicated as being about internally consistent counterfactuals, which we’ll label Raw Counterfactuals. When there are multiple raw counterfactuals consistent with your state of knowledge, you can pick the one with the highest utility.
However, there will also be cases where only a single counterfactual is consistent with your state of knowledge, which results in a rather trivial problem. Consider for example Transparent Newcomb’s Problem, where a perfect predictor places the million in a transparent box if and only if it predicts that you will one-box if it does. If you see the million, you know that you must have one-boxed so it doesn’t strictly make sense to ask what you should do in this situation. Instead, we’ll have to ask something slightly different instead. So, I’ve slightly modified my position since writing the Co-operation Game: in some situations logical counterfactuals will be defined relative to an imagined, rather than actual epistemic state. We will construct these states by erasing some information as described later in the post.
Other degenerative cases include when you already know what decision you’ll make or when you have the ability to figure it out. For example, when you have perfect knowledge of the environment and the agent, unless you run into issues with unprovability. Note that degeneracy is more common than you might think since knowing, for example, that it is a utility maximiser, tells you its exact behaviour in situations without options that are tied. Again, in these cases, the answer to the question, “What should the agent do?” is, “The only action consistent with the problem statement”. However, as we’ll see, it is possible sometimes to make these questions less trivial if you’re willing to accept a slight tweak to the problem statement.
We’ll consider two kinds of problems, acknowledging that these aren’t the only types. In external problems, we imagine decisions from the perspective of a theoretical, unbounded, non-embedded observer who exists outside of the problem statement. Clearly we can’t fully adopt the perspective of such an external agent, but describing the high-level details will usually suffice. Critically, in the external perspective, the observer can have goals, such as choosing the agent with the maximum utility, without those being the goals of the agent within the problem.
In (fully) reflective problems, we imagine decisions from the perspective of an agent considering its own decisions or potential decisions with full knowledge of its own source code. These problems will complicate the counterfactuals since the agent’s goals limit the kind of agent that it could be. For example, an agent that wants to maximise utility should only search over the possibility space where it is a utility maximiser.
Making this distinction more explicit: An external problem would ask “What decision maximises utility?”, as opposed to a reflective problem which asks: “What decision maximises utility for a utility maximiser?”. This distinction will mainly be important here in terms of when it makes a problem trivial or not.
The external/reflective distinction is very similar to the difference between embedded and non-embedded problems, but external problems can include embedded agents, just from the perspective of a non-embedded agent. So we can do a surprising amount of our theorising from within the external perspective.
Raw Counterfactuals
Raw counterfactuals are produced as follows: Starting with the territorywe use some process to produce a causal model. We can then imagine constructing different models by switching out or altering parts of the model. These represent a coherent concept seperate from any discussion of decisions. In so far as we care about what could have been, we need to ultimately relate our claims to raw counterfactuals as inconsistent models could not have been.
Causal Decision Theory uses its own notion of counterfactuals, which we’ll term Decision Counterfactuals. These are created by performing world surgery on models of causal processes. Unlike raw counterfactuals, decision counterfactuals are inconsistent. Suppose that you actually defect in the Prisoner’s Dilemma, but we are considering the counterfactual where you cooperate. Up until the point of the decision you are the kind of person who defects, but when we arrive at the decision, you magically cooperate.
Decision counterfactuals are useful because they approximate raw counterfactuals. Performing world surgery all the way back in time requires would require a lot of work. In theoretical decision problems it is usually easy to imagine a raw counterfactual that would match the problem description and provide the same answer as the decision counterfactual. In practical decision problems, we don’t have the data to do this.
Unfortunately, this approximation breaks down when performing world surgery to make a counterfactual decision consistent with the past requires us to change an element of the environment that is important for the specific problem. For example, in Newcomb’s problem, changing your current decision requires changing your past self which involves changing a predictor which is considered part of the environment. In this case, it makes sense to fall back to raw counterfactuals and build a new decision theory on top.
Functional decision theory, as normally characterised, is closer to this ideal as world surgery isn’t just performed on your decision, but also on all decisions that subjunctively depend on you. This removes a bunch of inconsistencies, however, we’ve still introduced an inconsistency by assuming that f(x)=b when f(x) really equals a. The raw counterfactual approach provides a stronger foundation because it avoids this issue. However, since proof-based FDT is very effective at handling reflective problems, it would be worthwhile rebuilding it upon this new foundation.
Let’s consider Newcomb’s problem from the external perspective. The external observer is trying to maximise utility rather than the agent within the problem, so there is no restriction on whether the agent can one-box or two-box whilst being consistent with the problem statement. We can then immediately observe that if we use raw counterfactuals, the reward only depends on the agent’s ultimate decision and agents that one-box score better than those who don’t. Simple cases like this which allow multiple consistent counterfactuals don’t require erasure.
On the other hand, there are problems which only allow a single raw counterfactual and hence require us to tweak the problem to make it well-defined. Consider, for examples, Transparent Newcomb’s, where if you see money in the box, you know that you will receive exactly $1 million. Some people say this fails to account for the agent in the simulator, but it’s entirely possible that Omega may be able to figure out what action you will take based on high level reasoning, as opposed having to run a complete simulation of you. We’ll describe later a way of tweaking the problem statement into something that is both consistent and non-trivial.
General Approach to Logical Counterfactuals
We will now attempt to produce a more solid theory of external problems using FDT. This will allow us to interpret decision problems where only one decision is consistent with the problem statement in a non-trivial way.
FDT frames logical counterfactuals as “What would the world be like if f(x)=b instead of a?” which doesn’t strictly make sense as noted in the discussion on raw counterfactuals. Two points: a) I think it should be clear that this question only makes sense in terms of thinking of perturbations of the map and not as a direct claim about the territory (see map and territory). b) We’ll address this problem by proposing a different approach for foundations, which these proof-based approaches should ultimately be justified in terms of.
There are two possible paths towards a more consistent theory of logical counterfactuals for these situations. In both cases we interpret the question of what it would mean to change the output of a function as an informal description of a similar question that is actually well-defined. The first approach is see what consequences can be logically deduced from f(x)=b while implementing a strategy to prevent us from deducing incorrect statements from the inconsistencies. This is often done by playing chicken with the universe. We will term this a paraconsistent approach, even though it doesn’t explicitly make use of paraconsistent logic as it is paraconsistent in spirit.
An alternative approach would be to interpret this sentence as making claims about raw counterfactuals. In FDT terms, the raw counterfactual approach finds an f’ such that f’(x)=b and also with certain as of yet unstated similarities to f and substitute this into all subjunctively linked processes. The paraconsistent approach is easier to do informally, but I suspect that the raw counterfactually approach would be more amenable to formalisation and provides more philosophical insight into what is actually going on. In so far as the paraconsistent approach may be more convenient for an implementation perspective than the first, we can justify it by tying it to raw counterfactuals.
Decisions
Now that we’ve outlined the broad approach, we should dig more into the question of what exactly it means to make a decision. As I explained in a previous post, there’s a sense in which you don’t so much ‘make’ a decision as implement one. If you make something, it implies that it didn’t exist before and now it does. In the case of decisions, it nudges you towards believing that the decision you were going to implement wasn’t set, then you made a decision, and then it was. However, when “you” and the environment are defined down to the atom, you can only implement one decision. It was always the case from the start of time that you were going to implement that decision.
We note that if you have perfect information about the agent and the environment, you need to forget or at least pretend to forget some information about the agent so that we can produce counterfactual versions of the agent who decide slightly differently. See Shminux’s post on Logical Counterfactuals are Low Res for a similar argument, but framed slightly differently. The key difference is that I’m not suggesting just adding noise to the model, but forgetting specific information that doesn’t affect the outcome.
In Transparent Newcomb’s, it would be natural to erase the knowledge that the box is full. This would then result in two counterfactuals: a) the one where the agent sees an empty box and two boxes, b) the one where the agent sees a full box and one boxes. It would be natural to relax the scope of who we care about from the agent who sees a million in the box to the agent at the end of the problem regardless of what they see. If we do so, we then have a proper decision problem and we can see that one-boxing is better.
Actually there’s a slight hitch here. In order to define the outcome an agent receives, we need to define what the predictor will predict when an agent sees the box containing the million. But it is impossible to place a two-boxer in this situation. We can resolve this by defining the predictor as simulating the agent responding to an input representing an inconsistent situation as I’ve described in Counterfactuals for Perfect Predictors.
In order to imagine a consistent world, when we imagine a different “you”, we must also imagine the environment interacting with that different “you” so that, for example, the predictor makes a different prediction. Causal decision theorists construct these counterfactuals incorrectly and hence they believe that they can change their decision without changing the prediction. They fail to realise that they can’t actually “change” their decision as there is a single decision that they will inevitably implement. I suggest replacing “change a decision” with “shift counterfactuals” when it is important to be able to think clearly about these topics. It also clarifies why the prediction can change without backwards causation (my previous post on Newcomb’s problem contains further material on why this isn’t an issue).
Erasure
Here’s how the erasure may proceed for the example of Transparent’s Newcomb Problem. Suppose we erase all information about what decision the agent is going to make. This also requires erasing the fact that you see a million in the transparent box. Then we look at all counterfactually possible agents and notice that the reward depends only on whether you are an agent who ultimately one-boxes or an agent who two-boxes. Those who one-box see the million and then receive it, those who two-box see no money in the transparent box and receive $1000 only. The counterfactual involving one-boxing performs better than that involving two-boxing, so we endorse one-boxing.
Things become more complicated if we want to erase less information about the agent. For example, we might want an agent to know that it is a utility maximiser as this might be relevant to evaluating the outcome the agent will receive from future decisions. Suppose that if you one-box in Transparent Newcomb’s you’ll then be offered a choice of $1000 or $2000, but if you two-box you’ll be offered $0 or $10 million. We can’t naively erase all information about your decision process in order to compute a counterfactual of whether you should one-box or two-box. Otherwise, we end up with a situation where, for example, there are agents who one-box and get different rewards. Here the easiest solution is to “collapse” all of the decisions and ask about a policy instead that covers all three decisions that may be faced. Then we can calculate the expectations without producing any inconsistencies in the counterfactuals as it then becomes safe to erase the knowledge that it is a utility maximiser.
The 5 and 10 problem doesn’t occur with the forgetting approach. Compare: If you keep the belief that you are a utility maximiser then the only choice you can implement is 10 so we don’t have a decision problem. We can define all the possible strategies as follows, p probability of choosing 5 and 1-p probability of choosing 10, so forget everything about yourself except that you are one of these strategies. There’s no downside as there is no need for an agent to know whether or not it is a utility maximiser. So we can solve the 5 and 10 problem without epsilon exploration.
When will we be able to use this forgetting technique? One initially thought might be the same scope FDT is designed to be optimal on—problems where the reward only depends on your outputs or predicted outputs. Because only the output or predicted output matters and not the algorithm, these can be considered fair, unlike a problem where an alphabetical decision theorist (picks the first decision ordered in alphabetical order) is rewarded and every other type of agent is punished.
However, some problems where this condition doesn’t hold also seem fair. Like suppose there are long programs and short programs (in terms of running time). Further suppose programs can output either A or B. The reward is then determined purely based on these two factors. Now suppose that there exists a program that can calculate the utilities in each of these four worlds and then based upon this either terminate immediately or run for a long time and then output its choice of A or B. Assume that if it terminates immediately after the decision it’ll qualify as a short program, while if it terminates after a long time it is a long program. Then, as a first approximation, we can say that this is also a fair problem since it is possible to win in all cases.
It’s actually slightly more complex than this. An AI usually doesn’t have to win on only one problem. Adding code to handle more situations will extend the running time and may prevent such an AI from always being able to choose the dominant option. An AI might also want to do things like double check calculations or consider whether it is actually running on consistent outcomes, so winning the problem might put limitations on the AI in other ways.
But nonetheless, so long as we can write such a program that picks the best option, we can call the problem “fair” in a limited sense. It’s possible to extend our definition of “fair” further. Like suppose that it’s impossible to analyse all the options and still return in a short amount of time. This isn’t a problem if the maximum utility is in a long option.
In regards to running time, we can also end up with a non-binary notion of “fair” according to how much extra processing a program can squeeze in before having to decide the short option. This limits the ability of the AI to check/recheck its work and examine its philosophical assumptions before having to make the decision.
Final Thoughts
Logical counterfactuals are often framed in such a way that it seems that we should be building a model of subjunctive dependence directly from the atoms in the universe. Instead we produce these from causal model that identifies the current agent and a model of forgetting. This makes our job much easier as it allows us to disentangle this problem from many of the fundamental issues in ontology and philosophy of science.
Comparing the method in this post to playing chicken with the universe: using raw counterfactuals clarifies that such proof-based methods are simply tricks that allow us to act as though we’ve forgotten without forgetting.
In general, I would suggest that logical counterfactuals are about knowing which information to erase such that you can produce perfectly consistent counterfactuals. Further, I would suggest that if you can’t find information to erase that produces perfectly consistent counterfactuals then you don’t have a decision theory problem. Future work could explore exactly when this is possible and general techniques for making this work as the current exploration has been mainly informal.
Notes:
I changed my terminology from Point Counterfactuals to Decision Counterfactuals and Timeless Counterfactuals to Raw Counterfactuals in this post as this better highlights where these models come from.
This post was written with the support of the EA Hotel