If you make an agent by sticking together cut-off Solomonoff induction and e.g. causal decision theory, I do indeed buy that this agent will have problems. Because causal decision theory has problems.
But how serious will these problems be? What if you encrypt the agent’s thoughts, add pain sensors, and make a few other simple patches to deal with embeddedness?
I wouldn’t be comfortable handing the lightcone over to such a thing, but I don’t really expect it to fall over anytime soon.
If you make an agent by sticking together cut-off Solomonoff induction and e.g. causal decision theory, I do indeed buy that this agent will have problems. Because causal decision theory has problems.
But how serious will these problems be? What if you encrypt the agent’s thoughts, add pain sensors, and make a few other simple patches to deal with embeddedness?
I wouldn’t be comfortable handing the lightcone over to such a thing, but I don’t really expect it to fall over anytime soon.