I think it might be useful to consider the framing of being an embedded agent in a deterministic world (in Laplace’s demon sense). There is no primitive “should”, only an emergent one. The question to ask in that setup is “what kind of embedded agents succeed, according to their internal definition of success?” For example it is perfectly rational to believe in God in a setup in a situation where this belief improves your odds of success, for some internal definition of success. If one’s internal definition of success is different, fighting religious dogma might count as a path to success, or success in itself. This gets complicated in a hurry, since an agent’s internal definition of success changes in time and is not necessarily unique or coherent for a single agent. The whole concept of agency is an emergent one, as well, and is not always applicable. But the framework of embedded agency is a good start and useful grounding when one gets lost in complications.
I think it might be useful to consider the framing of being an embedded agent in a deterministic world (in Laplace’s demon sense). There is no primitive “should”, only an emergent one. The question to ask in that setup is “what kind of embedded agents succeed, according to their internal definition of success?” For example it is perfectly rational to believe in God in a setup in a situation where this belief improves your odds of success, for some internal definition of success. If one’s internal definition of success is different, fighting religious dogma might count as a path to success, or success in itself. This gets complicated in a hurry, since an agent’s internal definition of success changes in time and is not necessarily unique or coherent for a single agent. The whole concept of agency is an emergent one, as well, and is not always applicable. But the framework of embedded agency is a good start and useful grounding when one gets lost in complications.