It’s not clear to me that the decision-algorithm-chooser problem is more compact than the decision-algorithm problem.
Perhaps they’re not different problems. If we solve the decision problem and then the algorithm just replicates itself when we tell it to decide between decision algorithms, this suggests that the philosophy “behind” was actually not behind at all. But one way or the other, you don’t want to be stuck with the problem of trying to choose according to a higher criterion using only human wisdom. A quining algorithm could plausibly represent having hit bottom. But if there are leftover dilemmas then you haven’t hit bottom. Then you want to step back and try to compact further.
ETA: Do you have the intuition that philosophy can be formalized, and in a compact way?
I have an intuition that the functional parts can be formalized in a compact way, and that the rest is our own confusion.
Perhaps they’re not different problems. If we solve the decision problem and then the algorithm just replicates itself when we tell it to decide between decision algorithms, this suggests that the philosophy “behind” was actually not behind at all. But one way or the other, you don’t want to be stuck with the problem of trying to choose according to a higher criterion using only human wisdom.
The set of quining algorithms is infinite, right? Restricting to the set of quining algorithms rules out many pathological cases, but using human wisdom to choose between them is still unavoidable.
But if there are leftover dilemmas then you haven’t hit bottom. Then you want to step back and try to compact further.
How will you know when there are really no leftover dilemmas? In other words, how can you tell that a candidate quining decision algorithm is free of errors? For example, you once thought that it was harmless for a decision algorithm to use the Solomonoff prior (which assumes that the universe is computable). (I hope I’ve convinced you that’s an error.) Such a decision algorithm can certainly be quining, so don’t we need a way to prevent this type of error in the future?
I have an intuition that the functional parts can be formalized in a compact way, and that the rest is our own confusion.
Human philosophy is error prone, but also error tolerant and correcting in a way that no decision algorithm that anyone has ever proposed is. If you give a typical decision algorithm the wrong prior, utility function, or implicit theory of fairness, it will happily go on and do its thing without complaint. Restricting to the set of quining algorithms doesn’t solve this problem. Human beings can somehow sense such errors and try to correct them. I think this ability needs to be understood and programmed into an AI before it can be considered safe.
Perhaps they’re not different problems. If we solve the decision problem and then the algorithm just replicates itself when we tell it to decide between decision algorithms, this suggests that the philosophy “behind” was actually not behind at all. But one way or the other, you don’t want to be stuck with the problem of trying to choose according to a higher criterion using only human wisdom. A quining algorithm could plausibly represent having hit bottom. But if there are leftover dilemmas then you haven’t hit bottom. Then you want to step back and try to compact further.
I have an intuition that the functional parts can be formalized in a compact way, and that the rest is our own confusion.
The set of quining algorithms is infinite, right? Restricting to the set of quining algorithms rules out many pathological cases, but using human wisdom to choose between them is still unavoidable.
How will you know when there are really no leftover dilemmas? In other words, how can you tell that a candidate quining decision algorithm is free of errors? For example, you once thought that it was harmless for a decision algorithm to use the Solomonoff prior (which assumes that the universe is computable). (I hope I’ve convinced you that’s an error.) Such a decision algorithm can certainly be quining, so don’t we need a way to prevent this type of error in the future?
Human philosophy is error prone, but also error tolerant and correcting in a way that no decision algorithm that anyone has ever proposed is. If you give a typical decision algorithm the wrong prior, utility function, or implicit theory of fairness, it will happily go on and do its thing without complaint. Restricting to the set of quining algorithms doesn’t solve this problem. Human beings can somehow sense such errors and try to correct them. I think this ability needs to be understood and programmed into an AI before it can be considered safe.
Basic paradigm on which I operate: http://lesswrong.com/lw/l9/artificial_addition/