By the same logic of Quantum Immortality, shouldn’t we expect never to fall asleep, since we can’t observe ourselves while being asleep?
Heighn
Commenting on the first myth, Yudkowsky himself seems to be pretty sure of this when reading his comment here: http://econlog.econlib.org/archives/2016/03/so_far_my_respo.html. I know Yudkowsky’s post is written after this LessWrong article, but it still seems relevant to mention.
Agreed, especially when compared to http://www.fhi.ox.ac.uk/gcr-report.pdf.
Although, now that I think about it, this survey is about risks before 2100, so the 5% risk of superintelligent AI might be that low because some of the responders belief such AI not to happen before 2100. Still, it seems in sharp contrast with Yudkowsky’s estimate.
I wrote a response to this critique here: https://medium.com/how-to-build-an-asi/a-defense-of-functional-decision-theory-d86a9a19a755. I’m happy to receive feedback!
Thanks! I see it now, weird. No idea why that link doesn’t work, but crossposting indeed seems like a better idea. So thanks! I’ll do that instead.
I wrote a response here: https://www.lesswrong.com/posts/R8muGSShCXZEnuEi6/a-defense-of-functional-decision-theory, where I attempt to refute some of the points made here.
The question is not which action to take. The question is which decision theory gives the most utility. Any candidate for “best decision theory” should take the left box. This results in a virtually guaranteed save of $100 - and yes, a death burn in an extremely unlikely scenario. In that unlikely scenario, yes, taking the right box gives the most utility—but that’s answering the wrong question.
In the Newcomb case, there’s a disagreement about whether one-boxing can actually somehow cause there to be a million dollars in the box; CDT denies this possibility (because it takes no account of sufficiently accurate predictors), while timeless/logical/functional/whatever decision theories accept it.
To be clear, FDT does not accept causation that happens backwards in time. It’s not claiming that the action of one-boxing itself causes there to be a million dollars in the box. It’s the agent’s algorithm, and, further down the causal diagram, Omega’s simulation of this algorithm that causes the million dollars. The causation happens before the prediction and is nothing special in that sense.
Can you explain point 1 further, please? It seems to me subjunctive dependence happens regardless of note inclusion, and thus one’s decision theory should left-box in both cases. (I’ll respond to your other points as well.)
And if I select FDT, I would be the one “smiling from atop a heap of utility” in (10^24 − 1) out of 10^24 worlds.
But that’s not the case here. Here, you’ve learned that taking the Left box kills you, but you still have a choice! You can still choose to take Right! And live!
Yes, but the point is to construct a decision theory that recommends actions in a way that maximizes expected utility. Recommending left-boxing does that, because it saves you $100 in virtually every world. That’s it, really. You keep focusing on that 1 out of 10^24 possibility were you burn to death, but that doesn’t take anything away from FDT. Like I said: it’s not about which action to take, let alone which action in such an improbable scenario. It’s about what decision theory we need.
Yes, FDT insists that actually, you must choose in advance (by “choosing your algorithm” or what have you), and must stick to the choice no matter what. But that is a feature of FDT, it is not a feature of the scenario! The scenario does not require that you stick to your choice. You’re free to take Right and live, no matter what your decision theory says.
Well, I’d say FDT recognizes that you do choose in advance, because you are predictable. Apparently you have an algorithm running that makes these choices, and the predictor simulates that algorithm. It’s not that you “must” stick to your choice. It’s about constructing a theory that consistently recommends the actions that maximize expected utility.
I know I keep repeating that—but it seems that’s where our disagreement lies. You look at which action is best in a specific scenario, I look at what decision theory produces the most utility. An artificial superintelligence running a decision theory can’t choose freely no matter what the decision theory says: running the decision theory means doing what it says.
So you say. But in the scenario (and in any situation we actually find ourselves in), only the one, actual, world is available for inspection. In that actual world, I’m the one with the heap of utility, and you’re dead.
No. We can inspect more worlds. We know what happens given the agent’s choice and the predictor’s prediction. There are multiple paths, each with its own probability. The problem description focuses on that one world, yes. But the point remains—we need a decision theory, we need it to recommend an action (left-boxing or right-boxing), and left-boxing gives the most utility if we consider the bigger picture.
Totally, and the decision theory we need is one that doesn’t make such terrible missteps!
Do you agree that recommending left-boxing before the predictor makes its prediction is rational?
As to inspection, maybe I’m not familiar enough with the terminology there.
Re your last point: I was just thinking about that too. And strangely enough I missed that the boxes are open. But wouldn’t the note be useless in that case?
I will think about this more, but it seems to me your decision theory can’t recommend “Left-box, unless you see a bomb in left.”, and FDT doesn’t do this. The problem is, in that case the prediction influences what you end up doing. What if the predictor is malevolent, and predicts you choose right, placing the bomb in left? It could make you lose $100 easily. Maybe if you believed the predictor to be benevolent?
Well, uh… that is rather an important aspect of the scenario…
Sure. But given the note, I had the knowledge needed already, it seems. But whatever.
Indeed, the predictor doesn’t have to predict anything to make me lose $100; it can just place the bomb in the left box, period. This then boils down to a simple threat: “pay $100 or die!”. Hardly a tricky decision theory problem…
Didn’t say it was a tricky decision problem. My point was that your strategy is easily exploitable and may therefore not be a good strategy.
The problem description explicitly states the predictor doesn’t do that, so no.
Hey, thanks! That’s awesome.
You don’t see how the problem description preventing it is relevant?
The description doesn’t prevent malevolence, but it does prevent putting a bomb in left if the agent left-boxes.
I was thinking about this post and thought up the following experiment. Suppose, by some quantum mechanism, Bob has a 50% probability of falling asleep for the next 8 hours and a 50% probability of staying awake for the next 8 hours. By the same logic as QI, should Bob expect (with 100% certainty) to be awake after 2 hours, since he cannot observe himself being asleep? I would say no. But then, doesn’t QI fail as a result?