There’s nothing unusual about my assumptions regarding instrumental rationality. It’s just standard expected utility theory.
The place I see to object is with my way of spreading probabilities over Sia’s desires. But if you object to that, I want to hear more about which probably distribution I should be using to understand the claim that Sia’s desires are likely to rationalise power-seeking, resource acquisition, and so on. I reached for the most natural way of distributing probabilities I could come up with—I was trying to be charitable to the thesis, & interpreting it in light of the orthogonality thesis. But if that’s not the right way to distribute probability over potential desires, if it’s not the right way of understanding the thesis, then I’d like to hear something about what the right way of understanding it is.
Thanks for the read and for the response.
>None of your models even include actions that are analogous to the convergent actions on that list.
I’m not entirely sure what you mean by “model”, but from your use in the penultimate paragraph, I believe you’re talking about a particular decision scenario Sia could find herself in. If so, then my goal wasn’t to prove anything about a particular model, but rather to prove things about every model.
>The non-sequential theoretical model is irrelevant to instrumental convergence, because instrumental convergence is about putting yourself in a better position to pursue your goals later on.
Sure. I started with the easy cases to get the main ideas out. Section 4 then showed how those initial results extend to the case of sequential decision making.
>Section 4 deals with sequential decisions, but for some reason mainly gets distracted by a Newcomb-like problem, which seems irrelevant to instrumental convergence. I don’t see why you didn’t just remove Newcomb-like situations from the model?
I used the Newcomb problem to explain the distinction between sophisticated and resolute choice. I wasn’t assuming that Sia was going to be facing a Newcomb problem. I just wanted to help the reader understand the distinction. The distinction is important, because it makes a difference to how Sia will choose. If she’s a resolute chooser, then sequential decisions reduce to a single non-sequential decisions. She just chooses a contingency plan at the start, and then sticks to that contingency plan. Whereas if she’s a sophisticated chooser, then she’ll make a series of non-sequential decisions. In both cases, it’s important to understand how she’ll choose in non-sequential decisions, which is why I started off thinking about that in section 3.
>It seems clear to me that for the vast majority of the random utility functions, it’s very valuable to have more control over the future world state. So most sampled agents will take the instrumentally convergent actions early in the game and use the additional power later on.
I am not at all confident about what would happen with randomly sampled desires in this decision. But I am confident about what I’ve proven, namely: if she’s a resolute chooser with randomly sampled desires, then for any two contingency plans, Sia is just as likely to prefer the first to the second as she is to prefer the second to the first.
When it comes to the ‘power-seeking’ contingency plans, there are two competing biases. On the one hand, Sia is somewhat biased towards them for the simple reason that there are more of them. If some early action affords more choices later on, then there are going to be more contingency plans which make that early choice. On the other hand, Sia is somewhat biased against them, since they are somewhat less predictable—they leave more up to chance.
I’ve no idea which of these biases will win out in your particular decision. It strikes me as a pretty difficult question.