Aside from more general issues that have been previously addressed (Friendly AI is a pipe dream, and the big advances on philosophical questions have for the most part been made by methods other than philosophy), a couple of specifics:
We were selected for the ability to tell stories and win political arguments, and it seems to me that minds so selected, should be expected to be able to do philosophy, albeit not terribly well—which is indeed the case.
You criticize the universal prior because it would disagree with our intuition when presented with an alleged halting oracle (because the universal prior takes for granted that the universe is computable, whereas to human intuition this is an open question). On the one hand I have sympathy with your position, because while I would like to think the universe is computable, I also regard it as an open question. On the other hand… do you have any reason other than intuition to believe, in that scenario, our intuition would be right and the universal prior would be wrong?
the big advances on philosophical questions have for the most part been made by methods other than philosophy.
That is a really good point. Psychology, for instance, is old enough that much Psychological theory was developed before more rigorous empirical standards were developed. These early psych. theories were formulated primarily through philosophical methods (introspection, metaphorical/associative reasoning, etc.) They were very unsuccessful when they were eventually held up to empirical standards.
Aside from more general issues that have been previously addressed (Friendly AI is a pipe dream, and the big advances on philosophical questions have for the most part been made by methods other than philosophy), a couple of specifics:
We were selected for the ability to tell stories and win political arguments, and it seems to me that minds so selected, should be expected to be able to do philosophy, albeit not terribly well—which is indeed the case.
You criticize the universal prior because it would disagree with our intuition when presented with an alleged halting oracle (because the universal prior takes for granted that the universe is computable, whereas to human intuition this is an open question). On the one hand I have sympathy with your position, because while I would like to think the universe is computable, I also regard it as an open question. On the other hand… do you have any reason other than intuition to believe, in that scenario, our intuition would be right and the universal prior would be wrong?
That is a really good point. Psychology, for instance, is old enough that much Psychological theory was developed before more rigorous empirical standards were developed. These early psych. theories were formulated primarily through philosophical methods (introspection, metaphorical/associative reasoning, etc.) They were very unsuccessful when they were eventually held up to empirical standards.