This is a helpful tip, thank you. I’ll put Halpern on the to-read-eventually list, and it sounds like he’s probably required reading if I want to do a serious job on the alternatives-to-probabilistic-models post. In poking around, it looks like Halpern’s Reasoning about Knowledge might be an even better fit for the parts about world modeling that’s interoperable across agents, though I guess it comes at it from logic rather than from statistics, and from agreement axioms rather than from the environment.
(Mostly, though, I think I’ll stay aboard the probabilistic train; it still seems really promising to me and there’s so much to learn.)
I have not taken a fresh look at infrabayesianism since starting in on technical alignment research of my own. The last time I looked at it was pretty shallow; I listened to Vanessa on AXRP and I had a few conversations with her in person. I can’t say I have a strong opinion of it at the moment. Say more?
For a little more technical details about IB: To me the idea is to have a way to compactly represent and manipulate an entire set of probability distributions. From that perspective: because of how you can easily maximize/minimize linear functions on convex sets (because the extremum will be some extreme point of the convex set), and since the function from probability distributions to expected value of utility is linear, you can easily find the worst case value on the convex set. For example, you can imagine storing in your head the three vertexes of a triangle, and then quickly finding what the worst case distribution from any part of the triangle would give you.
This lets you efficiently consider more while storing less, which in this view in the whole point.
You can instead take the view that you are ‘totally uncertain’ about which distribution is ‘true’, and so ‘should’ consider convex sets because you can mix any two distributions (e.g. 75% that you draw from distribution X, 25% that you draw from Y).
To me, this line of reasoning is suspect; but it still looks like IB is a good *computational tool*.
IB uses imprecise probability to get learning theoretic guarantees when the truth is outside the hypothesis class. It’s more convincing to me than most non-(purely)probabilistic frameworks. This sequence is the only (?) legible introduction: https://www.lesswrong.com/s/n7qFxakSnxGuvmYAX
This is a helpful tip, thank you. I’ll put Halpern on the to-read-eventually list, and it sounds like he’s probably required reading if I want to do a serious job on the alternatives-to-probabilistic-models post. In poking around, it looks like Halpern’s Reasoning about Knowledge might be an even better fit for the parts about world modeling that’s interoperable across agents, though I guess it comes at it from logic rather than from statistics, and from agreement axioms rather than from the environment.
(Mostly, though, I think I’ll stay aboard the probabilistic train; it still seems really promising to me and there’s so much to learn.)
I have not taken a fresh look at infrabayesianism since starting in on technical alignment research of my own. The last time I looked at it was pretty shallow; I listened to Vanessa on AXRP and I had a few conversations with her in person. I can’t say I have a strong opinion of it at the moment. Say more?
For a little more technical details about IB: To me the idea is to have a way to compactly represent and manipulate an entire set of probability distributions. From that perspective: because of how you can easily maximize/minimize linear functions on convex sets (because the extremum will be some extreme point of the convex set), and since the function from probability distributions to expected value of utility is linear, you can easily find the worst case value on the convex set. For example, you can imagine storing in your head the three vertexes of a triangle, and then quickly finding what the worst case distribution from any part of the triangle would give you.
This lets you efficiently consider more while storing less, which in this view in the whole point.
You can instead take the view that you are ‘totally uncertain’ about which distribution is ‘true’, and so ‘should’ consider convex sets because you can mix any two distributions (e.g. 75% that you draw from distribution X, 25% that you draw from Y).
To me, this line of reasoning is suspect; but it still looks like IB is a good *computational tool*.
IB uses imprecise probability to get learning theoretic guarantees when the truth is outside the hypothesis class. It’s more convincing to me than most non-(purely)probabilistic frameworks. This sequence is the only (?) legible introduction: https://www.lesswrong.com/s/n7qFxakSnxGuvmYAX