Second, FDT doesn’t always get you the most utility. For example, consider the following exotic possible world: the actual world. In this one, if you hang around academic philosophers, they will think you’re silly if you adopt FDT. This will make you sad. So adopting FDT gets you less utility. Additionally, in the actual world, I would get less utility if I were an FDTer, because I find it fun to argue with FDTers about decision-theory. Or imagine that the government passed a law where they tortured everyone who thought FDT was the right view. FDTers wouldn’t be better off.
What? This is a bad argument, because this doesn’t depend on the decision theory in question at all. For any decision theory XDT, it is possible to construct a world where Omega gives you one bazillion utilons if you don’t follow XDT, and murders you if you do follow XDT. This is part of why these problems are called “unfair”. The point of FDT is that for fair problems like Parfit’s hitchhiker (isomorphic to transparent Newcomb and similar to William MacAskill’s version of Bomb! but not as contrived) FDT wins.
I’m going to spend some time on Parfit’s hitchhiker because it illustrates the issue with EDT + CDT: they can’t commit to anything. I claim that lots of problems like Parfit’s hitchhiker come up in real life all the time. Blackmail is just evil Parfit’s hitchhiker. Lots of employment situations are Parfit’s hitchhiker: someone might hire you if and only if they think you’ll actually do the work (and have limited recourse to stop you if not). FDT is the only decision theoretical framework which lets you commit to anything at all.[1]
Yes there are non-decision theoretic frameworks which do something like commitment (virtue-ish ethics or deontology) but these aren’t mathematically formulated.
As to your point that FDT isn’t well-defined mathematically yet… uhh… yeah, everyone knows that. That’s one of the main points of the logical uncertainty research agenda. That’s why thousands of keystrokes have been spilled over Lob’s theorem. There are lots of Lobian obstacles to get around when thinking about thinking. It’s possible that something like Logical Inductors (which can handle logical uncertainty) can solve FDT if given the right series of inputs, but I don’t know.
I’m aware that you endorse giving in to blackmail in extremis, which I disagree with as a general position (and I would definitely caution against posting it publicly on the internet).
it illustrates the issue with EDT + CDT: they can’t commit to anything
What does this mean? Of course an agent who endorses EDT or CDT can commit to things — commitments are actions they decide between, like anything else.
In this case “commitment” means something specific.
Suppose you are a selfish CDT agent, and I am considering whether to hire you to clean my house. Once you’re inside my house, you might steal my stuff instead of cleaning my house. Suppose that California Labour Laws require that I pay you up-front and I know I have no chance of getting my money or stuff back. Say your preference order is “Steal” > “Do the job” > “Don’t get hired”
You, before being hired, might say “Oh JB, I promise not to steal, please give me this job” but, once they’re inside the house, the only causal effect they can have on the outcome is steal or don’t steal. And since CDT only considers utilities downstream from each individual decision at any point in time, CDT will always steal. A CDT-operating agent is incapable of committing not to steal from me, in this case.
Therefore I will not hire you to clean my house, and you get minimal utility.
An FDT agent reasons thusly: suppose FDT endorses stealing. In that universe, JB knows this and does not hire me, so I do not get hired and get minimal utility. If FDT endorses doing the job, JB knows this and does hire me, so I do get hired and then do the job. Therefore I will do the job.
Therefore I will hire an FDT agent to clean my house, and the FDT agent will get the middling utility.
You’re stipulating that CDT-me in your thought experiment doesn’t have access to any (psychological) actions that causally bind me to not steal from you. Right? Then sure, CDT-me would steal if he ended up in your house, and you’d want to prevent this.
But you’re also stipulating that I do have access to the action “decide to follow FDT”. That’s something that would causally bind me to not steal from you, if I took it before you made your decision whether to hire me. Why is this action a legitimate option in the hypothetical, while various other non-FDT ways of binding oneself aren’t?
Fair point, and CDT agents self-modifying is a thing that has been studied. You might e.g. modify yourself to specifically not prefer the stealing in this case. My understanding is that these modifications are equivalent to the agent changing its decision theory: since the modifications that an agent chooses are predictable based on the decision properties of the scenarios it finds itself in, they can themselves be captured by a description of a decision procedure, which is just what a decision theory is.
I think the resulting decision theory is called son-of-CDT, and is mostly like FDT but not quite in certain circumstances. But this is deep MIRI knowledge which I’m not sure is actually published and I’m entirely going off of what I’ve seen ex-MIRIans post on LessWrong.
For any decision theory XDT, it is possible to construct a world where Omega gives you one bazillion utilons if you don’t follow XDT, and murders you if you do follow XDT. This is part of why these problems are called “unfair”.
That’s like saying that the Halting Problem isn’t an issue because problems that involve self-reference are unfair. You can’t just avoid the Halting Problem by saying “no explicit self-reference”, because seemingly reasonable stipulations that don’t explicitly have self-reference in them may imply it anyway.
It may turn out that for some decision theories, reasonable-seeming problems that don’t explicitly say “Omega punishes you if you follow XDT” may be equivalent to “Omega punishes you if you follow XDT” anyway.
That’s like saying that the Halting Problem isn’t an issue because problems that involve self-reference are unfair. You can’t just avoid the Halting Problem by saying “no explicit self-reference”, because seemingly reasonable stipulations that don’t explicitly have self-reference in them may imply it anyway.
If people hadn’t done roughly this, we would never have gotten the entire field of verifiable programs. Likewise, no-free-lunch theorems provide evidence that no brain can ever exist, and similar impossibility results show that GPT-style language models cannot exist (it’s impossible to learn the rules to a formal language from only positive examples of that language).
Ruling out a class of things as “unfair” or “unrealistic” is sometimes necessary. For the same reason that Godel’s incompleteness theorem shouldn’t stop you doing maths.
FDT is good because it works in the (relatively) un-contrived Newcomb’s problem which is equivalent to the totally un-contrived Parfit’s Hitchhiker problem. If you want to extend decision theory to problems where the agent is deceived, or is punished because of following a particular decision theory, you find that you need to do some mixture of
Accept more machinery, e.g. priors over possible settings, to account for deception, or P(you are in a world where FDTers get directly killed by Omega) vs P(you are in a world where CDTers get directly killed by Omega) and put together the computational complexity of different Omega scenarios and hash out the Solomonoff induction
Give up and have no decision theory be better than any other
If you want to extend decision theory to problems where the agent is deceived, or is punished because of following a particular decision theory,
The issue is that there may not be a choice. If you don’t want to extend decision theory to that kind of problem, exactly what are you going to do to exclude problems like that? It won’t necessarily have a line “if decision_theory == ‘XDT’” in it. It may not be very obvious, and it may not even be possible to determine, that some problem falls into a category that you want to exclude.
No, what I mean is that there’s a symmetry between the setup “Omega kills you if you follow FDT for the crime of following FDT” and the setup “Omega kills you if you follow CDT for the crime of following CDT” which isn’t necessarily present in setups like “Omega simulates you and then based on your actions in the simulation, does X or Y”. There’s other problems with the second kind of system in some cases if Omega is allowed to lie, since this also allows symmetry into the system.
You can set up a version of Newcomb’s problem where Omega never lies, but you can’t do that for e.g. Newcomb’s revenge, since in that case, Omega has to tell it’s simulation of you that you’re in the regular Newcomb’s problem.
Actually, the halting problem (well, its generalization, Rice’s theorem) allow you to get a more precise intuition for why punishing agents iff they follow XDT is ‘unfair’ (it would be Turing-uncomputable for Omega to decide if an agent follow XDT, even with his omniscience and infinite compute).
Lots of employment situations are Parfit’s hitchhiker: someone might hire you if and only if they think you’ll actually do the work (and have limited recourse to stop you if not). FDT is the only decision theoretical framework which lets you commit to anything at all.
This reminds me of the ultimatum game in that academics are stumped by it, but regular people have no problem navigating it—people will usually offer fair splits and reject unfair splits, thus reaping rewards in defiance of psychologists’ protests that this is “irrational”.
(Now that I’m thinking about it, the ultimatum game might be isomorphic to Parfit’s hitchhiker in the way that matters for discriminating between decision theories? Rejecting an unfair split after it’s already been offered seems analogous to paying the hitchhiker even after you’ve already been given a ride)
I don’t think the CDTers claim to Newcomb being unfair a valid one. The parallel claim is Omega saying “OK, you follow CDT, I am going to kill you.” which is a separate problem from Newcomblike problems where you’re being simulated.
And I don’t think your in-principle reasons actually hold up in practice. FDT’s output can be understood as “Output X such that, given the statement ‘FDT outputs X’, value is maximized” which in classical logic does break by the principle of explosion. This is, again, why the entire logical uncertainty agenda exists.
(My guess is that the answer is something like a modified FDT(N) outputting “Output X such that, if you feed ‘FDT outputs X’ into a logical inductor and run it for N steps, expected value is maximized” which provably[1] converge as to something we might call FDT)
A claim I would agree with is something like: “FDT as currently written down is not well-defined, but appears to be intuitively navigable and may be modifiable into something like F’DT which basically gets at the same flavour, using the language of logical uncertainty”? I get the impression your view is much stronger than that, along the lines of “FDT is so completely poorly-defined it’s pointless to ever look in that direction again, salt the earth.” based on the general vibe of this article, but that’s not justifiable from the evidence that we have!
“FDT outputs X” is importantly underspecified. If you just change what it says in your exact situation, then you have no account of how it changes the other algorithms like the simulator.
The bigger problem though isn’t the counterlogicals but analyzing how in the counterlogical world other algorithms would be different.
Newcomb’s Revenge actually is a fair problem which is not quite as much of a problem as “If you use FDT, I will kill you”. But if you accept Newcomb’s Revenge as symmetric to Newcomb’s problem then you have to also deny that any decision theory can be better than any other. For any decision problem where we simulate an agent in situation X, decide a payoff matrix based on X for a different instance which is in a different situation Y, then you can come up with a flip. Newcomb’s problem X is important because it depends on your behaviour in the situation that you are actually in! Which breaks the asymmetry.
And ok if you actually want to think about this in depth it gets super cursed super quickly, because any FDT agent will have to have a prior over being in a Newcomblike problem vs a Revengelike problem and if it turns out that in the real world, Newcomb problems are rare and Revenge problems are common, then FDT will update accordingly and start one-boxing, but actually the concept of high-fidelity simulation at some level itself breaks down because the simulator can induce any behaviour in your simulated self by giving you arbitrary inputs.
Newcomb’s problem also works if you treat the simluated agents as agents themselves who have an accurate prior over their own situation (the maths takes a while to shake out but it definitely does). While Revenge only works in this way if the simulated agents think they’re being simulated by a Newcomblike Omega (otherwise, if they know they’re 50:50 real or being simulated by a Revengelike Omega, they will two box).
I don’t understand your claim on how exactly the counterlogicals break. Normally the obstacle is Lob’s theorem, which IIUC Logical Induction fixes, but if you have a stronger argument then I would like to hear it.
What? This is a bad argument, because this doesn’t depend on the decision theory in question at all. For any decision theory XDT, it is possible to construct a world where Omega gives you one bazillion utilons if you don’t follow XDT, and murders you if you do follow XDT. This is part of why these problems are called “unfair”. The point of FDT is that for fair problems like Parfit’s hitchhiker (isomorphic to transparent Newcomb and similar to William MacAskill’s version of Bomb! but not as contrived) FDT wins.
I’m going to spend some time on Parfit’s hitchhiker because it illustrates the issue with EDT + CDT: they can’t commit to anything. I claim that lots of problems like Parfit’s hitchhiker come up in real life all the time. Blackmail is just evil Parfit’s hitchhiker. Lots of employment situations are Parfit’s hitchhiker: someone might hire you if and only if they think you’ll actually do the work (and have limited recourse to stop you if not). FDT is the only decision theoretical framework which lets you commit to anything at all.[1]
Yes there are non-decision theoretic frameworks which do something like commitment (virtue-ish ethics or deontology) but these aren’t mathematically formulated.
As to your point that FDT isn’t well-defined mathematically yet… uhh… yeah, everyone knows that. That’s one of the main points of the logical uncertainty research agenda. That’s why thousands of keystrokes have been spilled over Lob’s theorem. There are lots of Lobian obstacles to get around when thinking about thinking. It’s possible that something like Logical Inductors (which can handle logical uncertainty) can solve FDT if given the right series of inputs, but I don’t know.
I’m aware that you endorse giving in to blackmail in extremis, which I disagree with as a general position (and I would definitely caution against posting it publicly on the internet).
What does this mean? Of course an agent who endorses EDT or CDT can commit to things — commitments are actions they decide between, like anything else.
In this case “commitment” means something specific.
Suppose you are a selfish CDT agent, and I am considering whether to hire you to clean my house. Once you’re inside my house, you might steal my stuff instead of cleaning my house. Suppose that California Labour Laws require that I pay you up-front and I know I have no chance of getting my money or stuff back. Say your preference order is “Steal” > “Do the job” > “Don’t get hired”
You, before being hired, might say “Oh JB, I promise not to steal, please give me this job” but, once they’re inside the house, the only causal effect they can have on the outcome is steal or don’t steal. And since CDT only considers utilities downstream from each individual decision at any point in time, CDT will always steal. A CDT-operating agent is incapable of committing not to steal from me, in this case.
Therefore I will not hire you to clean my house, and you get minimal utility.
An FDT agent reasons thusly: suppose FDT endorses stealing. In that universe, JB knows this and does not hire me, so I do not get hired and get minimal utility. If FDT endorses doing the job, JB knows this and does hire me, so I do get hired and then do the job. Therefore I will do the job.
Therefore I will hire an FDT agent to clean my house, and the FDT agent will get the middling utility.
You’re stipulating that CDT-me in your thought experiment doesn’t have access to any (psychological) actions that causally bind me to not steal from you. Right? Then sure, CDT-me would steal if he ended up in your house, and you’d want to prevent this.
But you’re also stipulating that I do have access to the action “decide to follow FDT”. That’s something that would causally bind me to not steal from you, if I took it before you made your decision whether to hire me. Why is this action a legitimate option in the hypothetical, while various other non-FDT ways of binding oneself aren’t?
Fair point, and CDT agents self-modifying is a thing that has been studied. You might e.g. modify yourself to specifically not prefer the stealing in this case. My understanding is that these modifications are equivalent to the agent changing its decision theory: since the modifications that an agent chooses are predictable based on the decision properties of the scenarios it finds itself in, they can themselves be captured by a description of a decision procedure, which is just what a decision theory is.
I think the resulting decision theory is called son-of-CDT, and is mostly like FDT but not quite in certain circumstances. But this is deep MIRI knowledge which I’m not sure is actually published and I’m entirely going off of what I’ve seen ex-MIRIans post on LessWrong.
That’s like saying that the Halting Problem isn’t an issue because problems that involve self-reference are unfair. You can’t just avoid the Halting Problem by saying “no explicit self-reference”, because seemingly reasonable stipulations that don’t explicitly have self-reference in them may imply it anyway.
It may turn out that for some decision theories, reasonable-seeming problems that don’t explicitly say “Omega punishes you if you follow XDT” may be equivalent to “Omega punishes you if you follow XDT” anyway.
If people hadn’t done roughly this, we would never have gotten the entire field of verifiable programs. Likewise, no-free-lunch theorems provide evidence that no brain can ever exist, and similar impossibility results show that GPT-style language models cannot exist (it’s impossible to learn the rules to a formal language from only positive examples of that language).
Ruling out a class of things as “unfair” or “unrealistic” is sometimes necessary. For the same reason that Godel’s incompleteness theorem shouldn’t stop you doing maths.
FDT is good because it works in the (relatively) un-contrived Newcomb’s problem which is equivalent to the totally un-contrived Parfit’s Hitchhiker problem. If you want to extend decision theory to problems where the agent is deceived, or is punished because of following a particular decision theory, you find that you need to do some mixture of
Accept more machinery, e.g. priors over possible settings, to account for deception, or P(you are in a world where FDTers get directly killed by Omega) vs P(you are in a world where CDTers get directly killed by Omega) and put together the computational complexity of different Omega scenarios and hash out the Solomonoff induction
Give up and have no decision theory be better than any other
The issue is that there may not be a choice. If you don’t want to extend decision theory to that kind of problem, exactly what are you going to do to exclude problems like that? It won’t necessarily have a line “if decision_theory == ‘XDT’” in it. It may not be very obvious, and it may not even be possible to determine, that some problem falls into a category that you want to exclude.
No, what I mean is that there’s a symmetry between the setup “Omega kills you if you follow FDT for the crime of following FDT” and the setup “Omega kills you if you follow CDT for the crime of following CDT” which isn’t necessarily present in setups like “Omega simulates you and then based on your actions in the simulation, does X or Y”. There’s other problems with the second kind of system in some cases if Omega is allowed to lie, since this also allows symmetry into the system.
You can set up a version of Newcomb’s problem where Omega never lies, but you can’t do that for e.g. Newcomb’s revenge, since in that case, Omega has to tell it’s simulation of you that you’re in the regular Newcomb’s problem.
Actually, the halting problem (well, its generalization, Rice’s theorem) allow you to get a more precise intuition for why punishing agents iff they follow XDT is ‘unfair’ (it would be Turing-uncomputable for Omega to decide if an agent follow XDT, even with his omniscience and infinite compute).
This reminds me of the ultimatum game in that academics are stumped by it, but regular people have no problem navigating it—people will usually offer fair splits and reject unfair splits, thus reaping rewards in defiance of psychologists’ protests that this is “irrational”.
(Now that I’m thinking about it, the ultimatum game might be isomorphic to Parfit’s hitchhiker in the way that matters for discriminating between decision theories? Rejecting an unfair split after it’s already been offered seems analogous to paying the hitchhiker even after you’ve already been given a ride)
I address that. The point is, CDTers have a parallel claim to Newcomb’s problem being unfair.
The point isn’t just that FDT isn’t well-defined mathematically. I explain in-principle reasons why it can’t be.
I don’t think the CDTers claim to Newcomb being unfair a valid one. The parallel claim is Omega saying “OK, you follow CDT, I am going to kill you.” which is a separate problem from Newcomblike problems where you’re being simulated.
And I don’t think your in-principle reasons actually hold up in practice. FDT’s output can be understood as “Output X such that, given the statement ‘FDT outputs X’, value is maximized” which in classical logic does break by the principle of explosion. This is, again, why the entire logical uncertainty agenda exists.
(My guess is that the answer is something like a modified FDT(N) outputting “Output X such that, if you feed ‘FDT outputs X’ into a logical inductor and run it for N steps, expected value is maximized” which provably[1] converge as
A claim I would agree with is something like: “FDT as currently written down is not well-defined, but appears to be intuitively navigable and may be modifiable into something like F’DT which basically gets at the same flavour, using the language of logical uncertainty”? I get the impression your view is much stronger than that, along the lines of “FDT is so completely poorly-defined it’s pointless to ever look in that direction again, salt the earth.” based on the general vibe of this article, but that’s not justifiable from the evidence that we have!
I don’t have a proof but this is the general kind of thing that logical inductors prove.
Why is Newcomb’s any less fair than Newcomb’s revenge? https://www.umsu.de/blog/2022/772
“FDT outputs X” is importantly underspecified. If you just change what it says in your exact situation, then you have no account of how it changes the other algorithms like the simulator.
The bigger problem though isn’t the counterlogicals but analyzing how in the counterlogical world other algorithms would be different.
Newcomb’s Revenge actually is a fair problem which is not quite as much of a problem as “If you use FDT, I will kill you”. But if you accept Newcomb’s Revenge as symmetric to Newcomb’s problem then you have to also deny that any decision theory can be better than any other. For any decision problem where we simulate an agent in situation X, decide a payoff matrix based on X for a different instance which is in a different situation Y, then you can come up with a flip. Newcomb’s problem X is important because it depends on your behaviour in the situation that you are actually in! Which breaks the asymmetry.
And ok if you actually want to think about this in depth it gets super cursed super quickly, because any FDT agent will have to have a prior over being in a Newcomblike problem vs a Revengelike problem and if it turns out that in the real world, Newcomb problems are rare and Revenge problems are common, then FDT will update accordingly and start one-boxing, but actually the concept of high-fidelity simulation at some level itself breaks down because the simulator can induce any behaviour in your simulated self by giving you arbitrary inputs.
Newcomb’s problem also works if you treat the simluated agents as agents themselves who have an accurate prior over their own situation (the maths takes a while to shake out but it definitely does). While Revenge only works in this way if the simulated agents think they’re being simulated by a Newcomblike Omega (otherwise, if they know they’re 50:50 real or being simulated by a Revengelike Omega, they will two box).
I don’t understand your claim on how exactly the counterlogicals break. Normally the obstacle is Lob’s theorem, which IIUC Logical Induction fixes, but if you have a stronger argument then I would like to hear it.