Planned summary for the previous post for the Alignment Newsletter:
Consider a setting in which an agent can play a game against a predictor. The agent can choose to say zero or one. It gets 3 utility if it says something different from the predictor, and −1 utility if it says the same thing. If the predictor is near-perfect, but the agent models itself as having access to unpredictable randomness, then the agent will continually try to randomize (which it calculates has expected utility 1), and will continually lose.
Planned summary for this post:
The problem with the previous agent is that it never learns that it has the wrong causal model. If the agent is able to learn a better causal model from experience, then it can learn that it is not actually able to use unpredictable randomness, and so it will no longer expect a 50% chance of winning, and it will stop playing the game.
If the predictor is near-perfect, but the agent models itself as having access to unpredictable randomness, then the agent will continually try to randomize (which it calculates has expected utility 1), and will continually lose.
It’s actually worse than that for CDT; the agent is not actually trying to randomise, it is compelled to model the predictor as a process that is completely disconnected from its own actions, so it can freely pick the action that the predictor is least likely to pick—according to the CDT’s modelling of it. Or pick zero in the case of a tie. So the CDT agent is actually deterministic, and even if you gave it a source of randomness, it wouldn’t see any need to use it.
The problem with the previous agent is that it never learns that it has the wrong causal model. If the agent is able to learn a better causal model from experience, then it can learn that it is not actually able to use unpredictable randomness, and so it will no longer expect a 50% chance of winning, and it will stop playing the game.
[...] then it can learn that the predictor can actually predict the agent successfully, and so will no longer expect a 50% [...]
If the predictor is near-perfect, but the agent models its actions as independent of the predictor (since the prediction was made in the past), then the agent will have some belief about the prediction and will choose the less likely action for expected utility at least 1, and will continually lose.
The problem with the previous agent is that it never learns that it has the wrong causal model. If the agent is able to learn a better causal model from experience, then it can learn that the predictor can actually predict the agent successfully, and so will no longer expect a 50% chance of winning, and it will stop playing the game.
What you wrote is good, and not worth changing. But I wanted to mention that CDT is even more bonkers than that: the prediction can be made in the future, just as long as there is no causal path to how the predictor is predicting. In some cases, the predictor can even know the action taken, and still predict in a way that CDT thinks is causally disconnected.
You can also model the agent as failing to learn that its “unpredictable randomness” isn’t. It’s still the case that the simple analysis of “agents which can’t learn a true fact will fail in cases where that something matters” is good enough.
Planned summary for the previous post for the Alignment Newsletter:
Planned summary for this post:
It’s actually worse than that for CDT; the agent is not actually trying to randomise, it is compelled to model the predictor as a process that is completely disconnected from its own actions, so it can freely pick the action that the predictor is least likely to pick—according to the CDT’s modelling of it. Or pick zero in the case of a tie. So the CDT agent is actually deterministic, and even if you gave it a source of randomness, it wouldn’t see any need to use it.
[...] then it can learn that the predictor can actually predict the agent successfully, and so will no longer expect a 50% [...]
Thanks! I changed it to:
What you wrote is good, and not worth changing. But I wanted to mention that CDT is even more bonkers than that: the prediction can be made in the future, just as long as there is no causal path to how the predictor is predicting. In some cases, the predictor can even know the action taken, and still predict in a way that CDT thinks is causally disconnected.
You can also model the agent as failing to learn that its “unpredictable randomness” isn’t. It’s still the case that the simple analysis of “agents which can’t learn a true fact will fail in cases where that something matters” is good enough.