The calculator computes “What is 2 + 3?”, not “What does this calculator compute as the result of 2 + 3?” The answer to the former question is 5, but if the calculator were to ask the latter question instead, the result could self-consistently be anything at all! If the calculator returned 42, then indeed, “What does this calculator compute as the result of 2 + 3?” would in fact be 42.
That’s why you don’t let your calculator be sentient. FAI might give a number that makes you most happy, which might well be 42 if you are not relying on this number for anything useful. (E.g. it might tell 42 as a joke, knowing that you know what 2+3 is)
Edit: you can, however, have some requirements on the calculator’s output, and then there will be the number that satisfies those criteria; the x substitution will work to solve for this value, and in principle even to solve for protective measures to take against cosmic rays, and so on.
edit: and on the NDT, it doesn’t one-way substitute at start. It assumes equivalence.
Sure, if it happened to be in a situation where the most valuable thing to do by my standards was make me happy. Agreed. You seem to be implying that I should prefer to avoid this result… do you in fact believe that? If so, can you clarify why?
A somewhat analogous real-world situation: one of Siri’s possible responses to “open the pod bay doors” as a command is “We’re never going to live that down, are we?” This delights me enormously, and costs nothing of consequence. Should I prefer that this result be eliminated?
Actually I misunderstood his point with calculator. He was speaking of NDT with issues resulting from equivalence, i thought he was speaking of issues resulting from substitution. I did not mean to imply that you should avoid this result, simply that if you want your calculator to work by the decision theory I thought he was referring to, it got to have some utilities associated with outputs. And this doesn’t really help make a calculating device.
Who said anything about sentience? NDT, as described, is a perfectly comprehensible program that (in certain games that you or I would regard as fair tests) generates spurious counterfactuals and thus makes terrible decisions, thanks to a particular kind of circularity.
In this sequence, I’m not talking about FAI or anything beyond my current understanding, and I’m not intentionally drawing metaphors. I’m simply outlining programs which (if I could write a good automated theorem-prover) I could write myself, and comparing how they do in a straightforward tournament setting, with the twist of allowing read-access to source codes. We should be able to agree on that base level.
I mean this sort of circularity:
I agree that some forms are benign. The Naive Decision Theory post and cousin_it’s followup illustrate a malignant form.
That’s why you don’t let your calculator be sentient. FAI might give a number that makes you most happy, which might well be 42 if you are not relying on this number for anything useful. (E.g. it might tell 42 as a joke, knowing that you know what 2+3 is)
Edit: you can, however, have some requirements on the calculator’s output, and then there will be the number that satisfies those criteria; the x substitution will work to solve for this value, and in principle even to solve for protective measures to take against cosmic rays, and so on.
edit: and on the NDT, it doesn’t one-way substitute at start. It assumes equivalence.
Sure, if it happened to be in a situation where the most valuable thing to do by my standards was make me happy. Agreed.
You seem to be implying that I should prefer to avoid this result… do you in fact believe that?
If so, can you clarify why?
A somewhat analogous real-world situation: one of Siri’s possible responses to “open the pod bay doors” as a command is “We’re never going to live that down, are we?” This delights me enormously, and costs nothing of consequence. Should I prefer that this result be eliminated?
Actually I misunderstood his point with calculator. He was speaking of NDT with issues resulting from equivalence, i thought he was speaking of issues resulting from substitution. I did not mean to imply that you should avoid this result, simply that if you want your calculator to work by the decision theory I thought he was referring to, it got to have some utilities associated with outputs. And this doesn’t really help make a calculating device.
Gotcha. Thanks for clarifying.
Who said anything about sentience? NDT, as described, is a perfectly comprehensible program that (in certain games that you or I would regard as fair tests) generates spurious counterfactuals and thus makes terrible decisions, thanks to a particular kind of circularity.
In this sequence, I’m not talking about FAI or anything beyond my current understanding, and I’m not intentionally drawing metaphors. I’m simply outlining programs which (if I could write a good automated theorem-prover) I could write myself, and comparing how they do in a straightforward tournament setting, with the twist of allowing read-access to source codes. We should be able to agree on that base level.
Yea, NDT is no good, agreed about that. That doesn’t so much results from substitution as from full blown two way equivalence.