The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent’s posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.
Also, there are entities that are impossible to distinguish from halting oracles using all the computational resources in the universe, which are not actually halting oracles.
Given this, then contra Wei Dai, I don’t know how any human attempting to internally implement Bayesian inference could possibly become convinced that a halting oracle exists.
The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent’s posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.
You lost a level of indirection in there; computing the output of an algorithm does not mean believing that the output of that algorithm is true or even plausible. So the agent will correctly predict what the human will say, and believe that the human is mistaken.
The level of indirection isn’t necessary: the Solomonoff agent’s distribution is a weighted mixture of the outputs of all possible Turing machines, weighted according to the posterior probability of that Turing machine being the one that is generating the observations. Any Turing machine that predicts that the putative halting oracle gets one wrong on a particular trial gets downweighted to zero when that fails to occur.
The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent’s posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.
That looks like the same position that Eliezer took, and I think I already refuted it. Let me know if you’ve read the one-logic thread and found my argument wrong or unconvincing.
The idea is that universal prior is really about observation-predicting algorithms that agents run, and not about prediction of what will happen in the world. So, for any agent that runs a given anticipation-defining algorithm and rewards/punishes the universal prior-based agent according to it, we have an anticipation-computing program that will obtain higher and higher probability in the universal prior-based agent.
This by the way again highlights the distinction between what will actually happen, and what a person anticipates—predictions are about capturing the concept of anticipation, an aspect of how people think, and are not about what in fact can happen.
The conclusion follows (I think) because the Solomonoff agent is computing the posterior probability of all algorithms, including the one that implements the same computation the human implements. So after updating, the Solomonoff agent’s posterior probability for that algorithm should be higher than that of any other algorithm, and it draws the same conclusion the human does.
Given this, then contra Wei Dai, I don’t know how any human attempting to internally implement Bayesian inference could possibly become convinced that a halting oracle exists.
You lost a level of indirection in there; computing the output of an algorithm does not mean believing that the output of that algorithm is true or even plausible. So the agent will correctly predict what the human will say, and believe that the human is mistaken.
The level of indirection isn’t necessary: the Solomonoff agent’s distribution is a weighted mixture of the outputs of all possible Turing machines, weighted according to the posterior probability of that Turing machine being the one that is generating the observations. Any Turing machine that predicts that the putative halting oracle gets one wrong on a particular trial gets downweighted to zero when that fails to occur.
That looks like the same position that Eliezer took, and I think I already refuted it. Let me know if you’ve read the one-logic thread and found my argument wrong or unconvincing.
The idea is that universal prior is really about observation-predicting algorithms that agents run, and not about prediction of what will happen in the world. So, for any agent that runs a given anticipation-defining algorithm and rewards/punishes the universal prior-based agent according to it, we have an anticipation-computing program that will obtain higher and higher probability in the universal prior-based agent.
This by the way again highlights the distinction between what will actually happen, and what a person anticipates—predictions are about capturing the concept of anticipation, an aspect of how people think, and are not about what in fact can happen.