To give a meta summary of the problems in this essay: the author does not define their terms, runs rampant with them, and is then shocked when they run into contradictory intuitions.
Here FDT’s answer is that you should two-box in the first case but not in the second case.
No, it says to two-box in both cases for exactly that axiom of extensionality (“equivalent prediction principle”).
No fact about whether two algorithms are the same
This entire objection is a failure to recognize that 0% and 100% are not probabilities. Rather than saying, “are these the exact same?” which is always impossible to verify to 100% confidence, not just with algorithms, you should ask, “how similar are these policies?” The goal of functional decision theory is to maximize the utility of all agents with similar policies to your own, weighted by exp(-KL(their policy||your policy)). So, for example, your twin that presents exactly the same except on Newcomb’s problem has the maximally distant policy to your own when you run into Newcomb.
The calculator objection is made up. You define your terms and then claim they are not defined! You gave us the isomorphism between the calculator’s algorithm and the neg-calculator’s algorithm. Under the axiom of extensionality, these are the same algorithms. Unless you want to define “algorithm” differently, which involves the minus sign. In which case, they’re slightly different algorithms (see the previous paragraph).
Determining how one algorithm being different would affect another algorithm being different, without depending on the epistemic probability of the second being different if the first was.
Is being exponentially accurate in polynomial time good enough? If so, just use annealing to find the trembling-hand equilibria. If not, you’re asking to solve P=NP.
The predictor has a failure rate of only 1 in a trillion trillion.
How? A rational decision-maker chooses mixed policies (for that entropy bonus), and even when there is certain, painful death on the table, they will almost certainly choose it more than 1 in a trillion trillion times. Even an irrational actor will simply mess up and walk in the wrong direction more often than 1 in a trillion trillion. If they claim they can predict my randomness ahead of time, they are lying. As a good functional decision theorist, I use a quantum random number generator (e.g. stare at a lightbulb while thinking) to prevent my randomness being hacked like that.
The simulation is highly correlated with you, so his guesses about whether you’ll cut off your leg for no benefit are 99.9% accurate.
The issue with most of these scenarios is you are unclear on what you mean by 99.9% accurate. Is this epistemic or aleatoric uncertainty? If it is epistemic, Newcomb is not powerful enough to change his decision based on your algorithm. Your algorithm should be: pretend to be a one-boxer (or leg-cutter) up until actually put in that scenario. If he gives you tests before putting you in the scenario, well an EDT or CDT would certainly do their best to pass the tests as well for the future utility.
If it is only the aleatoric uncertainty in your policy, we have set up as an axiom that Newcomb knows your policy. Then you object, “why not just change the policy?” But you literally just made a stipulation that you can’t! It’s exactly the failure most people make with the Grandfather’s Paradox. What seems to be possible is physically impossible when you impose axiomatic restrictions.
The goal of functional decision theory is to maximize the utility of all agents with similar policies to your own, weighted by exp(-KL(their policy||your policy)).
Huh, may I have a source on this? I thought you could point FDT at maximizing any utility function you like.
The issue with saying, “this agent,” is you do not actually know its policy. The best anyone can do is generate all programs that output the seen distribution of actions, using error-correction codes for nondeterministic policies. Now you have many theories of varying description lengths for the agent, which you weight according to the Solomonoff prior. We can always describe another agent’s policy with a fixed KL(their policy||your policy) extra error-correction bits, so the utils under a given theory are
Why do you need to know the policy in order to figure out the utility function? I thought you could point FDT at, like, maximizing Chaitin’s constant. I am hoping to look at whatever reference document you are getting your definitions from, is there no such thing?
There’s a lot of snark here but it’s all incorrect.
//No, it says to two-box in both cases for exactly that axiom of extensionality (“equivalent prediction principle”).//
I explain why this isn’t right. Only one algorithm is dependent on yours. The other is just correlated.
//This entire objection is a failure to recognize that 0% and 100% are not probabilities. Rather than saying, “are these the exact same?” which is always impossible to verify to 100% confidence, not just with algorithms, you should ask, “how similar are these policies?”//
This is wrong on a number of counts. First of all, measures of similarities are not the same as probabilities. So this doesn’t require any claim about similarity. Note that FDT wants to say that if you’re in prisoner’s dilemma against a perfect twin, your actions are correlated 100% with what they do (even if you don’t have a credence of 1 that that is so). Second, as I explain, measuring similarity is even more difficult.
Re calculator, whether they output the same thing depends on how you interpret their outputs, as explained in the post.
Re being able to determine what’s true of one algorithm from the other, that’s just looking at correlation which can’t be the relevant notion for the reasons I explain.
We imagine an agent who never messes up an accidentally picks the wrong one. By the lights of FDT, you get more expected utility timelessly if you’re always disposed not to pay. And it’s a stipulation of the thought experiment that the predictor is reliable—doesn’t matter if this could exist in the real world.
99.9% accurate in the sense that 99.9% of the time, the predictor guesses right. We additionally can imagine that his decision depends on what you do on the last moment, not just on what you’re pretending to do until then.
I acknowledge the snark. I get annoyed when people repeatedly make the mistake of not defining their terms, running rampant with them, being shocked when they get mismatched intuitions, and conclude the undefinitions wrong. It’s no more logical than, “you’re wrong because I feel that way.”
I explain why this isn’t right. Only one algorithm is dependent on yours. The other is just correlated.
Correlated is doing a lot of equivocating in your intuitions. It’s merely correlated not causal, he says! What’s the difference? everyone asks. Oh, there is none, they are extensionally identical, but using the word correlated will trick the functional decision theorist into taking a different action.
One man’s modus tollens is another man’s modus ponens.
You say, “since my intuitions imply the functional decision theorist will take different actions in these extensionally equivalent scenarios, clearly the functional decision theorist can be Dutch booked.”
I say, “since the functional decision theorist is rational and cannot be Dutch booked, clearly your intuitions are wrong about what the functional decision theorist will do. Go back and straighten out your definitions.”
Second, as I explain, measuring similarity is even more difficult.
I literally told you how to measure similarity: KL(p||q).
Re calculator, whether they output the same thing depends on how you interpret their outputs, as explained in the post.
I didn’t think this was your main objection because you told us the isomorphism. However, if the isomoprhism is unknown, or there isn’t an isomorphism but some other transformation, you can use the mutual information to recover it. Re: mutual information neural estimation.
99.9% accurate in the sense that 99.9% of the time, the predictor guesses right.
And it’s a stipulation of the thought experiment that the predictor is reliable—doesn’t matter if this could exist in the real world.
Ah yes, the principle of explosion proves every proposition true and false. Just sneak in contradictory axioms (by not defining your terms) and you can prove anything you want!
To give a meta summary of the problems in this essay: the author does not define their terms, runs rampant with them, and is then shocked when they run into contradictory intuitions.
No, it says to two-box in both cases for exactly that axiom of extensionality (“equivalent prediction principle”).
This entire objection is a failure to recognize that 0% and 100% are not probabilities. Rather than saying, “are these the exact same?” which is always impossible to verify to 100% confidence, not just with algorithms, you should ask, “how similar are these policies?” The goal of functional decision theory is to maximize the utility of all agents with similar policies to your own, weighted by exp(-KL(their policy||your policy)). So, for example, your twin that presents exactly the same except on Newcomb’s problem has the maximally distant policy to your own when you run into Newcomb.
The calculator objection is made up. You define your terms and then claim they are not defined! You gave us the isomorphism between the calculator’s algorithm and the neg-calculator’s algorithm. Under the axiom of extensionality, these are the same algorithms. Unless you want to define “algorithm” differently, which involves the minus sign. In which case, they’re slightly different algorithms (see the previous paragraph).
Is being exponentially accurate in polynomial time good enough? If so, just use annealing to find the trembling-hand equilibria. If not, you’re asking to solve P=NP.
How? A rational decision-maker chooses mixed policies (for that entropy bonus), and even when there is certain, painful death on the table, they will almost certainly choose it more than 1 in a trillion trillion times. Even an irrational actor will simply mess up and walk in the wrong direction more often than 1 in a trillion trillion. If they claim they can predict my randomness ahead of time, they are lying. As a good functional decision theorist, I use a quantum random number generator (e.g. stare at a lightbulb while thinking) to prevent my randomness being hacked like that.
The issue with most of these scenarios is you are unclear on what you mean by 99.9% accurate. Is this epistemic or aleatoric uncertainty? If it is epistemic, Newcomb is not powerful enough to change his decision based on your algorithm. Your algorithm should be: pretend to be a one-boxer (or leg-cutter) up until actually put in that scenario. If he gives you tests before putting you in the scenario, well an EDT or CDT would certainly do their best to pass the tests as well for the future utility.
If it is only the aleatoric uncertainty in your policy, we have set up as an axiom that Newcomb knows your policy. Then you object, “why not just change the policy?” But you literally just made a stipulation that you can’t! It’s exactly the failure most people make with the Grandfather’s Paradox. What seems to be possible is physically impossible when you impose axiomatic restrictions.
Huh, may I have a source on this? I thought you could point FDT at maximizing any utility function you like.
The issue with saying, “this agent,” is you do not actually know its policy. The best anyone can do is generate all programs that output the seen distribution of actions, using error-correction codes for nondeterministic policies. Now you have many theories of varying description lengths for the agent, which you weight according to the Solomonoff prior. We can always describe another agent’s policy with a fixed KL(their policy||your policy) extra error-correction bits, so the utils under a given theory are
sum_{policy} exp(-|theory| - KL(policy||your policy)) utils(policy)
and the total utils are
sum_{theory} sum_{policy} … = constant * sum_{policy} exp(-KL(policy||your policy) utils(policy)
I assume you mean Arithmetic coding.
Why do you need to know the policy in order to figure out the utility function? I thought you could point FDT at, like, maximizing Chaitin’s constant. I am hoping to look at whatever reference document you are getting your definitions from, is there no such thing?
There is no such thing.
There’s a lot of snark here but it’s all incorrect.
//No, it says to two-box in both cases for exactly that axiom of extensionality (“equivalent prediction principle”).//
I explain why this isn’t right. Only one algorithm is dependent on yours. The other is just correlated.
//This entire objection is a failure to recognize that 0% and 100% are not probabilities. Rather than saying, “are these the exact same?” which is always impossible to verify to 100% confidence, not just with algorithms, you should ask, “how similar are these policies?”//
This is wrong on a number of counts. First of all, measures of similarities are not the same as probabilities. So this doesn’t require any claim about similarity. Note that FDT wants to say that if you’re in prisoner’s dilemma against a perfect twin, your actions are correlated 100% with what they do (even if you don’t have a credence of 1 that that is so). Second, as I explain, measuring similarity is even more difficult.
Re calculator, whether they output the same thing depends on how you interpret their outputs, as explained in the post.
Re being able to determine what’s true of one algorithm from the other, that’s just looking at correlation which can’t be the relevant notion for the reasons I explain.
We imagine an agent who never messes up an accidentally picks the wrong one. By the lights of FDT, you get more expected utility timelessly if you’re always disposed not to pay. And it’s a stipulation of the thought experiment that the predictor is reliable—doesn’t matter if this could exist in the real world.
99.9% accurate in the sense that 99.9% of the time, the predictor guesses right. We additionally can imagine that his decision depends on what you do on the last moment, not just on what you’re pretending to do until then.
I acknowledge the snark. I get annoyed when people repeatedly make the mistake of not defining their terms, running rampant with them, being shocked when they get mismatched intuitions, and conclude the undefinitions wrong. It’s no more logical than, “you’re wrong because I feel that way.”
Correlated is doing a lot of equivocating in your intuitions. It’s merely correlated not causal, he says! What’s the difference? everyone asks. Oh, there is none, they are extensionally identical, but using the word correlated will trick the functional decision theorist into taking a different action.
One man’s modus tollens is another man’s modus ponens.
You say, “since my intuitions imply the functional decision theorist will take different actions in these extensionally equivalent scenarios, clearly the functional decision theorist can be Dutch booked.”
I say, “since the functional decision theorist is rational and cannot be Dutch booked, clearly your intuitions are wrong about what the functional decision theorist will do. Go back and straighten out your definitions.”
I literally told you how to measure similarity: KL(p||q).
I didn’t think this was your main objection because you told us the isomorphism. However, if the isomoprhism is unknown, or there isn’t an isomorphism but some other transformation, you can use the mutual information to recover it. Re: mutual information neural estimation.
Ah yes, the principle of explosion proves every proposition true and false. Just sneak in contradictory axioms (by not defining your terms) and you can prove anything you want!