algon33 comments on Comparing reward learning/​reward tampering formalisms

[algon33]

Stuart, by ”$P_{rt}$$\left(R|D_{1;j}\right)$ is complex” are you referring to their using $R=R(.,E[{\Theta }_{\ast }^R|D_{1;j}\left]\right)$ as the estimated reward function?

Also, what did you think of their arguement that their agents have no incentive to manipulate their beliefs because they evaluate future trajectories based of their current beliefs about how likely they are? Does that suffice to implement eq. 1) from your motivated value selection paper?

[Stuart_Armstrong]

Suart, by ”$P_{rt}(R\mid D_{1:j})$ is complex” are you referring to...

I mean that that defining $P_{rt}$ can be done in many different ways, and hence has a lot of contingent structure. In contrast, in $P_{lp}(R\mid D_{1:j},\rho )$, the $\rho is a complex distribution on $R$, conditional on $D_{1:j}$; hence $P_{lp}$ itself is trivial and just encodes “apply $\rho$ to $R$ and $D_{1:j}$ in the obvious way.