I mean that humans are born with more primitive versions of such drives (e.g. valuing others smiling as a proxy for compassion), then instrumental convergence transforms these primitive versions into the actual approval reward/status seeking/etc in a manner similar to training making agents who already are in the basin of alignment/corrigibility/etc more aligned/corrigible. As for prisoners no longer wanting to escape, this is where your conjecture about the belief system is most promising: a few unsuccessful attempts instill a hard-to-reassess belief that future attempts have a low probability of success.
UPD: how similar is the belief-desire model to the master-slave model? Suppose that the belief system simply outputs what it believes to be the next tokens of, say, known theorems or of muscular actions responsible for pull-ups and the desire system steers the belief system in order to construct the outcomes like solving the problem or doing the exercise correctly. Then the desire and belief system would occupy the positions of the master and the slave.
I mean that humans are born with more primitive versions of such drives (e.g. valuing others smiling as a proxy for compassion), then instrumental convergence transforms these primitive versions into the actual approval reward/status seeking/etc in a manner similar to training making agents who already are in the basin of alignment/corrigibility/etc more aligned/corrigible. As for prisoners no longer wanting to escape, this is where your conjecture about the belief system is most promising: a few unsuccessful attempts instill a hard-to-reassess belief that future attempts have a low probability of success.
UPD: how similar is the belief-desire model to the master-slave model? Suppose that the belief system simply outputs what it believes to be the next tokens of, say, known theorems or of muscular actions responsible for pull-ups and the desire system steers the belief system in order to construct the outcomes like solving the problem or doing the exercise correctly. Then the desire and belief system would occupy the positions of the master and the slave.