axioman comments on In theory: does building the subagent have an “impact”?

[axioman]

So the general problem is that large changes in $Q_R(s_{t+1},$∅) are not penalized?

[Stuart_Armstrong]

It’s the delta of that with $Q_R(s_{t+1},a_{t+1})$ that is penalised, not large changes on its own.