Here’s how I’d summarize my disagreement with the main claim: Alice is not acting rationally in your thought experiment if she acts like Bob (under some reasonable assumptions). In particular, she is doing pure exploitation and zero (value-)exploration by just maximizing her current weighted sum. For example, she should be reading philosophy papers.
See my reply to Rohin above—I wasn’t very clear about it in the OP, but I meant to consider questions where the AI knows no philosophy papers etc. are available.
Here’s how I’d summarize my disagreement with the main claim: Alice is not acting rationally in your thought experiment if she acts like Bob (under some reasonable assumptions). In particular, she is doing pure exploitation and zero (value-)exploration by just maximizing her current weighted sum. For example, she should be reading philosophy papers.
See my reply to Rohin above—I wasn’t very clear about it in the OP, but I meant to consider questions where the AI knows no philosophy papers etc. are available.