Yeah, what I’m doing here is more or less a formalisation of the ideas in your writeup, with the added technical complication that the “math intuition model” is nondeterministic so you need to use matrix counterfactuals. In order to get UDTish instead of CDTish behavior, I am going to make the agent select some sort of “logical policy” instead of action (i.e. something that reduces to a metathreat in a game theoretic setting).
Yeah, what I’m doing here is more or less a formalisation of the ideas in your writeup, with the added technical complication that the “math intuition model” is nondeterministic so you need to use matrix counterfactuals. In order to get UDTish instead of CDTish behavior, I am going to make the agent select some sort of “logical policy” instead of action (i.e. something that reduces to a metathreat in a game theoretic setting).