To the extent that I understand your models here, I suspect they don’t meaningfully bind/correspond to reality. (Of course, I don’t understand your models at all well, and I don’t have the energy to process the whole post, so this doesn’t really provide you with much evidence; sorry.)
I wonder how one could test whether or not the models bind to reality? E.g. maybe there are case examples (of agents/people behaving in instrumentally rational ways) one could look at, and see if the models postdict the actual outcomes in those examples?
There’s nothing unusual about my assumptions regarding instrumental rationality. It’s just standard expected utility theory.
The place I see to object is with my way of spreading probabilities over Sia’s desires. But if you object to that, I want to hear more about which probably distribution I should be using to understand the claim that Sia’s desires are likely to rationalise power-seeking, resource acquisition, and so on. I reached for the most natural way of distributing probabilities I could come up with—I was trying to be charitable to the thesis, & interpreting it in light of the orthogonality thesis. But if that’s not the right way to distribute probability over potential desires, if it’s not the right way of understanding the thesis, then I’d like to hear something about what the right way of understanding it is.
Thanks for the response.
To the extent that I understand your models here, I suspect they don’t meaningfully bind/correspond to reality. (Of course, I don’t understand your models at all well, and I don’t have the energy to process the whole post, so this doesn’t really provide you with much evidence; sorry.)
I wonder how one could test whether or not the models bind to reality? E.g. maybe there are case examples (of agents/people behaving in instrumentally rational ways) one could look at, and see if the models postdict the actual outcomes in those examples?
There’s nothing unusual about my assumptions regarding instrumental rationality. It’s just standard expected utility theory.
The place I see to object is with my way of spreading probabilities over Sia’s desires. But if you object to that, I want to hear more about which probably distribution I should be using to understand the claim that Sia’s desires are likely to rationalise power-seeking, resource acquisition, and so on. I reached for the most natural way of distributing probabilities I could come up with—I was trying to be charitable to the thesis, & interpreting it in light of the orthogonality thesis. But if that’s not the right way to distribute probability over potential desires, if it’s not the right way of understanding the thesis, then I’d like to hear something about what the right way of understanding it is.