I don’t see why it should necessarily undercut the core message of the post, since inner optimizers are still in some sense about the consequences of a pure predictive accuracy optimizer (but in the selection sense, not the control sense). But I agree that it wasn’t sufficiently well done. It didn’t feel like a natural next complication, the way everything else did.
I wouldn’t say that inner optimizers are about the consequences of pure predictive accuracy optimization; the two are orthogonal. An inner optimizer can pop up in optimizers which optimize for things besides predictive accuracy, and predictive accuracy optimization can be done in ways which don’t give rise to inner optimizers. Contrast that to the other failure modes discussed in the post, which are inherently about predictive accuracy—e.g. the assassination markets problem.
I don’t see why it should necessarily undercut the core message of the post, since inner optimizers are still in some sense about the consequences of a pure predictive accuracy optimizer (but in the selection sense, not the control sense). But I agree that it wasn’t sufficiently well done. It didn’t feel like a natural next complication, the way everything else did.
I wouldn’t say that inner optimizers are about the consequences of pure predictive accuracy optimization; the two are orthogonal. An inner optimizer can pop up in optimizers which optimize for things besides predictive accuracy, and predictive accuracy optimization can be done in ways which don’t give rise to inner optimizers. Contrast that to the other failure modes discussed in the post, which are inherently about predictive accuracy—e.g. the assassination markets problem.
OK, yeah, that’s fair.