My response to this is that the probability distribution is even less up for grabs.
Really? In practice I have a great deal of uncertainty about both my utility function and my probability estimates. Accurate probability estimates require the ability to accurately model the world, and this seems incredibly hard in general. It’s not at all clear to me that instrumental rationality means trusting your current probability estimates if you have reason to believe that future evidence will drastically change them or that they’re corrupted for some other reason (even an otherwise flawlessly designed AI has to worry about cosmic rays flipping the bits in its memory or, Omega forbid, its source code).
I am definitely not saying “trust your current probability estimates”.
What I’m saying is that probability should reflect reality as closely as possible, whereas utility should reflect preferences as closely as possible.
Modifying the preference function in an ad-hoc way to get the right behavior is a bad idea, but modifying our expectation about how reality actually might be is even worse. The probability function should be modified exclusively in response to considerations about how reality might be. The utility function should be modified exclusively in response to considerations about our preferences.
Really? In practice I have a great deal of uncertainty about both my utility function and my probability estimates. Accurate probability estimates require the ability to accurately model the world, and this seems incredibly hard in general. It’s not at all clear to me that instrumental rationality means trusting your current probability estimates if you have reason to believe that future evidence will drastically change them or that they’re corrupted for some other reason (even an otherwise flawlessly designed AI has to worry about cosmic rays flipping the bits in its memory or, Omega forbid, its source code).
I am definitely not saying “trust your current probability estimates”.
What I’m saying is that probability should reflect reality as closely as possible, whereas utility should reflect preferences as closely as possible.
Modifying the preference function in an ad-hoc way to get the right behavior is a bad idea, but modifying our expectation about how reality actually might be is even worse. The probability function should be modified exclusively in response to considerations about how reality might be. The utility function should be modified exclusively in response to considerations about our preferences.