Eliezer seems to think (or, at least he did at the time) that this isn’t a solvable problem. To phrase the question in a way more relevant to recent discussions, are those statements in any way similar to “a halting oracle exists”?
Solomonoff’s prior can’t predict something uncomputable, but I don’t see anything obviously uncomputable about any of the 3 statements you asked about.
Yes. Anything that can be represented by a turing machine gets a nonzero prior. And its model of itself goes in the same turing machine with the rest of the world.
“Where’d you get your universal prior, Neo?”
Eliezer seems to think (or, at least he did at the time) that this isn’t a solvable problem. To phrase the question in a way more relevant to recent discussions, are those statements in any way similar to “a halting oracle exists”?
Solomonoff’s prior can’t predict something uncomputable, but I don’t see anything obviously uncomputable about any of the 3 statements you asked about.
Right. But can it predict computable scenarios in which it is wrong?
Yes. Anything that can be represented by a turing machine gets a nonzero prior. And its model of itself goes in the same turing machine with the rest of the world.