I’m actually trying to be somewhat agnostic about the right conclusion here. I could have easily added another chapter discussing why the maximizing-surprise idea is not quite right. The moral is that the questions are quite complicated, and thinking vaguely about ‘optimization processes’ is quite far from adequate to understand this. Furthermore, it’ll depend quite a bit on the actual details of a training procedure!
I’m actually trying to be somewhat agnostic about the right conclusion here. I could have easily added another chapter discussing why the maximizing-surprise idea is not quite right. The moral is that the questions are quite complicated, and thinking vaguely about ‘optimization processes’ is quite far from adequate to understand this. Furthermore, it’ll depend quite a bit on the actual details of a training procedure!