Looks good! As you say, it is more technical than illuminating, but it’s probably a necessary step if the whole project is going to work out. I had definitely thought that this might be tricky, so it’s good to see it done. Also I think that there were some natural follow-up questions that I had set aside because I didn’t have this piece.
I don’t think we should really care about whether the theory proves useful facts about the distribution; as long as the distribution doesn’t dogmatically believe falsehoods about itself, we can just condition on explicit reflective observations and that will probably be fine.
So I’m basically fine with this diagonalization step, though I might come to share your concerns if I thought about it longer.
I think the key next step, that may or may not be possible, is getting a distribution that can assign non-zero probability to its own reflection principle while being reflective. This obviously requires weakening the reflection principle, though I think that there are some weak enough reflection principles floating around.
For example, it might be worth taking another look at the double expectation property (E[f(E)] = E[E[f(E)]] for functions f that are continuous in the product topology), and seeing what that translates into in this framework. Obviously continuity will be replaced by something different, since now E and f are both just Turing machines; hopefully we would get E[f] = E[E[f]] for all f that are approximable relative to some powerful oracle. It looks to me like this might work, and if the oracle was powerful enough to implement E I’d consider that promising as a potential basis for self-trust.
Looks good! As you say, it is more technical than illuminating, but it’s probably a necessary step if the whole project is going to work out. I had definitely thought that this might be tricky, so it’s good to see it done. Also I think that there were some natural follow-up questions that I had set aside because I didn’t have this piece.
I don’t think we should really care about whether the theory proves useful facts about the distribution; as long as the distribution doesn’t dogmatically believe falsehoods about itself, we can just condition on explicit reflective observations and that will probably be fine.
So I’m basically fine with this diagonalization step, though I might come to share your concerns if I thought about it longer.
I think the key next step, that may or may not be possible, is getting a distribution that can assign non-zero probability to its own reflection principle while being reflective. This obviously requires weakening the reflection principle, though I think that there are some weak enough reflection principles floating around.
For example, it might be worth taking another look at the double expectation property (E[f(E)] = E[E[f(E)]] for functions f that are continuous in the product topology), and seeing what that translates into in this framework. Obviously continuity will be replaced by something different, since now E and f are both just Turing machines; hopefully we would get E[f] = E[E[f]] for all f that are approximable relative to some powerful oracle. It looks to me like this might work, and if the oracle was powerful enough to implement E I’d consider that promising as a potential basis for self-trust.