I’m really happy to see this. I’ve had similar thoughts about the Good Regulator Theorem, but didn’t take the time to write them up or really pursue the fix.
Marginally related: my hope at some point was to fix the Good Regulator Theorem and then integrate it with other representation theorems, to come up with a representation theorem which derived several things simultaneously:
Probabilistic beliefs (or some appropriate generalization).
Expected utility theory (or some appropriate generalization).
A notion of “truth” based on map/territory correspondence (or some appropriate modification). This is the Good Regulator part.
The most ambitious version I can think of would address several more questions:
A. Some justification for the basic algebra. (I think sigma-algebras are probably not the right algebra to base beliefs on. Something resembling linear logic might be better for reasons we’ve discussed privately; that’s very speculative of course. Ideally the right algebra should be derived from considerations arising in construction of the representation theorem, rather than attempting to force any outcome top-down.) This is related to questions of what’s the right logic, and should touch on questions of constructivism vs platonism. Due to point #3, it should also touch on formal theories of truth, particularly if we can manage a theorem related to embedded agency rather than cartesian (dualistic) agency.
B. It should be better than CCT in that it should represent the full preference ordering, rather than only the optimal policy. This may or may not lead to InfraBayesian beliefs /expected values (the InfraBayesian representation theorem being a generalization of CCT which represents the whole preference ordering).
C. It should ideally deal with logical uncertainty, not just the logically omniscient case. This is hard. (But your representation theorem for logical induction is a start.) Or failing that, it should at least deal with a logically omniscient version of Radical Probabilism, ie address the radical-probabilist critique of Bayesian updating. (See my post Radical Probabilism; currently typing on a phone, so getting links isn’t convenient.
D. Obviously it would ideally also deal with questions of CDT vs EDT (ie present a solution to the problem of counterfactuals).
E. And deal with tiling problems, perhaps as part of the basic criteria.
I think sigma-algebras are probably not the right algebra to base beliefs on. Something resembling linear logic might be better for reasons we’ve discussed privately; that’s very speculative of course. Ideally the right algebra should be derived from considerations arising in construction of the representation theorem, rather than attempting to force any outcome top-down.
Have you elaborated on this somewhere or can you link some resource about why linear logic is a better algebra for beliefs than sigma algebra?
Not sure this is exactly what you meant by the full preference ordering, but might be of interest: I give the preorder of universally-shared-preferences between “models” here (in section 4).
Basically, it is the Blackwell order, if you extend the Blackwell setting to include a system.
I’m really happy to see this. I’ve had similar thoughts about the Good Regulator Theorem, but didn’t take the time to write them up or really pursue the fix.
Marginally related: my hope at some point was to fix the Good Regulator Theorem and then integrate it with other representation theorems, to come up with a representation theorem which derived several things simultaneously:
Probabilistic beliefs (or some appropriate generalization).
Expected utility theory (or some appropriate generalization).
A notion of “truth” based on map/territory correspondence (or some appropriate modification). This is the Good Regulator part.
The most ambitious version I can think of would address several more questions:
A. Some justification for the basic algebra. (I think sigma-algebras are probably not the right algebra to base beliefs on. Something resembling linear logic might be better for reasons we’ve discussed privately; that’s very speculative of course. Ideally the right algebra should be derived from considerations arising in construction of the representation theorem, rather than attempting to force any outcome top-down.) This is related to questions of what’s the right logic, and should touch on questions of constructivism vs platonism. Due to point #3, it should also touch on formal theories of truth, particularly if we can manage a theorem related to embedded agency rather than cartesian (dualistic) agency.
B. It should be better than CCT in that it should represent the full preference ordering, rather than only the optimal policy. This may or may not lead to InfraBayesian beliefs /expected values (the InfraBayesian representation theorem being a generalization of CCT which represents the whole preference ordering).
C. It should ideally deal with logical uncertainty, not just the logically omniscient case. This is hard. (But your representation theorem for logical induction is a start.) Or failing that, it should at least deal with a logically omniscient version of Radical Probabilism, ie address the radical-probabilist critique of Bayesian updating. (See my post Radical Probabilism; currently typing on a phone, so getting links isn’t convenient.
D. Obviously it would ideally also deal with questions of CDT vs EDT (ie present a solution to the problem of counterfactuals).
E. And deal with tiling problems, perhaps as part of the basic criteria.
Have you elaborated on this somewhere or can you link some resource about why linear logic is a better algebra for beliefs than sigma algebra?
I asked Abram about this live here https://youtu.be/zIC_YfLuzJ4?si=l5xbTCyXK9UhofIH&t=5546
Would you be able to share Abram’s answer in written form? I’d be keen to hear what he has to say.
Not sure this is exactly what you meant by the full preference ordering, but might be of interest: I give the preorder of universally-shared-preferences between “models” here (in section 4).
Basically, it is the Blackwell order, if you extend the Blackwell setting to include a system.