I think the process is hard to formalize because specifying step 2 seems to require specifying a decision theory almost directly. Recall that causal decision theorists argue that two-boxing is the right choice in Newcomb’s problem. Similarly, some would argue that not giving the money in counterfactual mugging is the right choice from the perspective of the agent who already knows that it lost, whereas others argue for the opposite. Or take a look at the comments on the Two-Boxing Gene. Generally, the kind of decision problems that put decision theories to a serious test also tend to be ones in which it is non-obvious what the right choice is. The same applies to meta-principles. Perhaps people agree with the vNM axioms, but desiderata that could shed a light on Newcomblike problems appear to be more controversial. For example, irrelevance of impossible outcomes and reflective stability both seem desirable but actually contradict each other.
TL;DR: It seems to be really hard to specify what it means for a decision procedure to “win”/fail in a given thought experiment.
I agree! I think that it this is hard for humans working with current syntactic machinary to specify things like:
* what their decision thoery will return for every decision problem
* what split(DT_1,DT_2) looks like
Right now I think doing this requires putting all decision theories on a useful shared ontology. The way that UTMs put all computable algorithms on a useful shared ontology which allowed people to make proofs about algorithms in general. This looks hard and possibly requires creating new kinds of math.
I am making the assumption here that the decision theories are rescued to the point of being executable philosophy. DTs need to be specified this much to be run by an AI. I believe that the fuzzy concepts inside people’s heads about how can in principle be made to work mathematically and then run on a computer. In a similar way that the fuzzy concept of “addition” was ported to symbolic representations and then circuits in a pocket calculator.
Interesting post! :)
I think the process is hard to formalize because specifying step 2 seems to require specifying a decision theory almost directly. Recall that causal decision theorists argue that two-boxing is the right choice in Newcomb’s problem. Similarly, some would argue that not giving the money in counterfactual mugging is the right choice from the perspective of the agent who already knows that it lost, whereas others argue for the opposite. Or take a look at the comments on the Two-Boxing Gene. Generally, the kind of decision problems that put decision theories to a serious test also tend to be ones in which it is non-obvious what the right choice is. The same applies to meta-principles. Perhaps people agree with the vNM axioms, but desiderata that could shed a light on Newcomblike problems appear to be more controversial. For example, irrelevance of impossible outcomes and reflective stability both seem desirable but actually contradict each other.
TL;DR: It seems to be really hard to specify what it means for a decision procedure to “win”/fail in a given thought experiment.
I agree! I think that it this is hard for humans working with current syntactic machinary to specify things like:
* what their decision thoery will return for every decision problem
* what split(DT_1,DT_2) looks like
Right now I think doing this requires putting all decision theories on a useful shared ontology. The way that UTMs put all computable algorithms on a useful shared ontology which allowed people to make proofs about algorithms in general. This looks hard and possibly requires creating new kinds of math.
I am making the assumption here that the decision theories are rescued to the point of being executable philosophy. DTs need to be specified this much to be run by an AI. I believe that the fuzzy concepts inside people’s heads about how can in principle be made to work mathematically and then run on a computer. In a similar way that the fuzzy concept of “addition” was ported to symbolic representations and then circuits in a pocket calculator.