Can I have a little more detail on the setup? Is it a fair restatement to say: You’re an agent, with a static reward function which you do not have direct access to. Omega (God, your creator, someone infallable and honest) has told you that 0.5R1 + 0.5R2 is reducable to your reward function, somehow, and you are not capable of experimenting or observing anything that would disambiguate this.
Now, as an actual person, I’d probably say “Fuck you, God, I’m running the experiment. I’ll do something that generates different R1 and R2, measure my reward, and now I know my weighting.”
In the case of an artificially-limited agent, who isn’t permitted to actually update based on experience, you’re right that it doesn’t matter—probability _is_ weight for uncertain outcomes. But you have an unnecessary indirection with “respects conservation of expected evidence. ” You can just say “unable to update this belief”.
Can I have a little more detail on the setup? Is it a fair restatement to say: You’re an agent, with a static reward function which you do not have direct access to. Omega (God, your creator, someone infallable and honest) has told you that 0.5R1 + 0.5R2 is reducable to your reward function, somehow, and you are not capable of experimenting or observing anything that would disambiguate this.
Now, as an actual person, I’d probably say “Fuck you, God, I’m running the experiment. I’ll do something that generates different R1 and R2, measure my reward, and now I know my weighting.”
In the case of an artificially-limited agent, who isn’t permitted to actually update based on experience, you’re right that it doesn’t matter—probability _is_ weight for uncertain outcomes. But you have an unnecessary indirection with “respects conservation of expected evidence. ” You can just say “unable to update this belief”.