I think the fact that traders are updating “behind the scenes” is an important problem with logical inductors (and with Solomonoff induction, though the logical inductors case is philosophically clearer to think about). It seems more natural to me to study that problem in the purely epistemic setting though.
In particular, there are conditions where we systematically expect traders to predict badly, e.g. because some of them are consequentialists and by predicting badly they can influence us in a desired way. As a result, although logical inductors are reflectively consistent in the limit, at finite times we don’t approximately trust their judgments (even after they have run for more than long enough to update on all of the logical facts that we know).
I am more interested in progress on this problem than about the application to decision theory (and I think that the epistemic version is equally philosophically appealing), so if I were thinking about thin priors I would have a somewhat different focus.
the notion of a good thin prior might be partially dependent on subjective human judgments, and so not amenable to math
I agree with this, but if we lower the bar from “correct” to not actively bad it feels like there ought to be a solution.
I agree that the epistemic formulation is probably more broadly useful, e.g. for informed oversight. The decision theory problem is additionally compelling to me because of the apparent paradox of having a changing caring measure. I naively think of the caring measure as fixed, but this is apparently impossible because, well, you have to learn logical facts. (This leads to thoughts like “maybe EU maximization is just wrong; you don’t maximize an approximation to your actual caring function”.)
I think the fact that traders are updating “behind the scenes” is an important problem with logical inductors (and with Solomonoff induction, though the logical inductors case is philosophically clearer to think about). It seems more natural to me to study that problem in the purely epistemic setting though.
In particular, there are conditions where we systematically expect traders to predict badly, e.g. because some of them are consequentialists and by predicting badly they can influence us in a desired way. As a result, although logical inductors are reflectively consistent in the limit, at finite times we don’t approximately trust their judgments (even after they have run for more than long enough to update on all of the logical facts that we know).
I am more interested in progress on this problem than about the application to decision theory (and I think that the epistemic version is equally philosophically appealing), so if I were thinking about thin priors I would have a somewhat different focus.
I agree with this, but if we lower the bar from “correct” to not actively bad it feels like there ought to be a solution.
I agree that the epistemic formulation is probably more broadly useful, e.g. for informed oversight. The decision theory problem is additionally compelling to me because of the apparent paradox of having a changing caring measure. I naively think of the caring measure as fixed, but this is apparently impossible because, well, you have to learn logical facts. (This leads to thoughts like “maybe EU maximization is just wrong; you don’t maximize an approximation to your actual caring function”.)