Huh, yeah, I suppose that side-effects from research could swing it. I’m not really sure how to analyze this.
Here’s a bit of an expanded line of thinking/intuition, following the same thread as I started on in my previous comment, but getting a bit less concrete and a bit more speculative. It’s an argument for why this kind of thing will (maybe) rule out some policies as impermissible:
In order for you to not suffer sure-losses, you must be representable as an EV-maximizing agent, which means that there must be some beliefs for which your entire policy (your mapping from all possible future observations, to all possible actions) is optimal w.r.t. total welfare.
(Of course, as bounded agents, even if we try hard to be EV-maximizing-w.r.t.-precise-credences, we’ll be suffering sure-losses as viewed from the perspective of epistemically superior agents. Some of our beliefs are contradictory, etc. So what we really want here is some milder version of this criterion that admits to the limitation of bounded agents, but still condemns inefficiencies that we are capable of avoiding.)
An intuitively compelling criterion is: these precise beliefs (which you are representable as holding) are within the bounds of your imprecise credences.
I’m not sure if this criterion follows naturally from dynamic strong maximality, or if it’s a plausible “extra criterion” to add on top, or if you think it’s a bad criterion that we shouldn’t assume.
For the rest of this comment, I’ll assume it.
But when I picture myself living a life under cluelessness about impartial long-run welfare, as I currently intuitively understand such lives to look like (doing stuff like “working on animal welfare”, or “caring for friends and family”, or “researching/advocating for cluelessness”), and in particular, when I picture the big complex object that is the policy of a person doing these sorts of actions — including all the ways they could react to all possible information — it’s really unclear whether even severly imprecise credences should reasonably include any set of beliefs under which that policy is optimal w.r.t. total welfare.
Intuitively, it seems like that might require oddly contorted beliefs. For example, for some reason, your actions change hugely when you learn about details of slaughter methods for shrimp, but change not at all when you learn what-seems-like-it-ought-to-be highly relevant information about the long-run consequences of your actions. Or perhaps your actions change a lot when you discover a new consideration for how to do reasoning under cluelessness. Are there really any “reasonable” precise beliefs (i.e. within the bounds of your imprecise credences) that prescribe that precise policy?
(One possible direction to explore: What’s the “size” of “the set of all reasonable precise credences”, and what’s the “size” of “the set of all possible policies”. If the former is vastly larger than the latter, then maybe it doesn’t seem so unlikely that there will always be some set of “reasonable credences” that recommends any particular policy. But if not, then constraining yourself to only follow policies that are recommended by some possible “reasonable credences” will probably significantly limit the set of policies that are permissible to choose.)
An intuitively compelling criterion is: these precise beliefs (which you are representable as holding) are within the bounds of your imprecise credences.
I think this is the step I reject. By hypothesis, I don’t think the coherence arguments show that the precise distribution P that I can be represented as optimizing w.r.t. corresponds to (reasonable) beliefs. P is nothing more than a mathematical device for representing some structure of behavior. So I’m not sure why I should require that my representor — i.e., the set of probability distributions that would be no less reasonable than each other if adopted as beliefs[1] — contains P.
I think I maybe figured out how to show that P must be in the representor.
You ought to assign non-0 probability that you’ll be asked to bet on arbitrary questions. In order to not have your policy be dominated, dynamic maximality will require that you commit in advance to the odds that you’d be on (after seeing arbitrary evidence). Clearly you should be on odds P. And it’s only permissible to bet at odds that are inside your representor.
(Now strictly speaking, there are some nuances about what kind of questions you can be convinced that you’ll be betting on, given that some of them might be quite hard to measure/verify even post-hoc. But since we’re just talking about non-0 probability of being convinced that you’re really betting on a question, I don’t think this should be too restrictive. And even non-”pure” bets, that only indirectly gets at some question q, will contribute to forcing P’s belief in q inside of your representor, I think.)
Sorry, I don’t understand the argument yet. Why is it clear that I should bet on odds P, e.g., if P is the distribution that the CCT says I should be represented by?
Because you couldn’t be represented as being an EV-maximizer with beliefs P if you were betting using some odds other than P. Because that would lead to lower expected value. (Assuming that pay-offs are going to be proportional to some proper scoring rule.)
“It’s only permissible to bet at odds that are inside your representor” is only true if the representor is convex. If my credence in some proposition X is, say, P(X) = (0.2, 0.49) U (0.51, 0.7), IIUC it’s permissible to bet at 0.5. I guess the claim that’s true is “It’s only permissible to bet at odds in the convex hull of your representor”.
But I’m not aware of an argument that representors should be convex in general.
If there is such an argument, my guess is that the way things would work is: We start with the non-convex set of distributions that seem no less reasonable than each other, and then add in whichever other distributions are needed to make it convex. But there would be no particular reason we’d need to interpret these other distributions as “reasonable” precise beliefs, relative to the distributions in the non-convex set we started with.
And, the kind of precise distribution P that would rationalize e.g. working on shrimp welfare seems to be the analogue of “betting at 0.5” in my example above. That is:
Our actual “set of distributions that seem no less reasonable than each other” would include some distributions that imply large positive long-term EV from working on shrimp welfare, and some that imply large negative long-term EV.
Whereas the distributions like P that imply vanishingly small long-term EV — given any evidence too weak to resolve our cluelessness w.r.t. long-term welfare — would lie in the convex hull. So betting at odds P would be permissible, and yet this wouldn’t imply that P is “reasonable” as precise beliefs.
Huh, yeah, I suppose that side-effects from research could swing it. I’m not really sure how to analyze this.
Here’s a bit of an expanded line of thinking/intuition, following the same thread as I started on in my previous comment, but getting a bit less concrete and a bit more speculative. It’s an argument for why this kind of thing will (maybe) rule out some policies as impermissible:
In order for you to not suffer sure-losses, you must be representable as an EV-maximizing agent, which means that there must be some beliefs for which your entire policy (your mapping from all possible future observations, to all possible actions) is optimal w.r.t. total welfare.
(Of course, as bounded agents, even if we try hard to be EV-maximizing-w.r.t.-precise-credences, we’ll be suffering sure-losses as viewed from the perspective of epistemically superior agents. Some of our beliefs are contradictory, etc. So what we really want here is some milder version of this criterion that admits to the limitation of bounded agents, but still condemns inefficiencies that we are capable of avoiding.)
An intuitively compelling criterion is: these precise beliefs (which you are representable as holding) are within the bounds of your imprecise credences.
I’m not sure if this criterion follows naturally from dynamic strong maximality, or if it’s a plausible “extra criterion” to add on top, or if you think it’s a bad criterion that we shouldn’t assume.
For the rest of this comment, I’ll assume it.
But when I picture myself living a life under cluelessness about impartial long-run welfare, as I currently intuitively understand such lives to look like (doing stuff like “working on animal welfare”, or “caring for friends and family”, or “researching/advocating for cluelessness”), and in particular, when I picture the big complex object that is the policy of a person doing these sorts of actions — including all the ways they could react to all possible information — it’s really unclear whether even severly imprecise credences should reasonably include any set of beliefs under which that policy is optimal w.r.t. total welfare.
Intuitively, it seems like that might require oddly contorted beliefs. For example, for some reason, your actions change hugely when you learn about details of slaughter methods for shrimp, but change not at all when you learn what-seems-like-it-ought-to-be highly relevant information about the long-run consequences of your actions. Or perhaps your actions change a lot when you discover a new consideration for how to do reasoning under cluelessness. Are there really any “reasonable” precise beliefs (i.e. within the bounds of your imprecise credences) that prescribe that precise policy?
(One possible direction to explore: What’s the “size” of “the set of all reasonable precise credences”, and what’s the “size” of “the set of all possible policies”. If the former is vastly larger than the latter, then maybe it doesn’t seem so unlikely that there will always be some set of “reasonable credences” that recommends any particular policy. But if not, then constraining yourself to only follow policies that are recommended by some possible “reasonable credences” will probably significantly limit the set of policies that are permissible to choose.)
Thanks for explaining!
I think this is the step I reject. By hypothesis, I don’t think the coherence arguments show that the precise distribution P that I can be represented as optimizing w.r.t. corresponds to (reasonable) beliefs. P is nothing more than a mathematical device for representing some structure of behavior. So I’m not sure why I should require that my representor — i.e., the set of probability distributions that would be no less reasonable than each other if adopted as beliefs[1] — contains P.
I’m not necessarily committed to this interpretation of the representor, but for the purposes of this discussion I think it’s sufficient.
I think I maybe figured out how to show that P must be in the representor.
You ought to assign non-0 probability that you’ll be asked to bet on arbitrary questions. In order to not have your policy be dominated, dynamic maximality will require that you commit in advance to the odds that you’d be on (after seeing arbitrary evidence). Clearly you should be on odds P. And it’s only permissible to bet at odds that are inside your representor.
(Now strictly speaking, there are some nuances about what kind of questions you can be convinced that you’ll be betting on, given that some of them might be quite hard to measure/verify even post-hoc. But since we’re just talking about non-0 probability of being convinced that you’re really betting on a question, I don’t think this should be too restrictive. And even non-”pure” bets, that only indirectly gets at some question q, will contribute to forcing P’s belief in q inside of your representor, I think.)
Sorry, I don’t understand the argument yet. Why is it clear that I should bet on odds P, e.g., if P is the distribution that the CCT says I should be represented by?
Because you couldn’t be represented as being an EV-maximizer with beliefs P if you were betting using some odds other than P. Because that would lead to lower expected value. (Assuming that pay-offs are going to be proportional to some proper scoring rule.)
Oops, right. I think what’s going on is:
“It’s only permissible to bet at odds that are inside your representor” is only true if the representor is convex. If my credence in some proposition X is, say, P(X) = (0.2, 0.49) U (0.51, 0.7), IIUC it’s permissible to bet at 0.5. I guess the claim that’s true is “It’s only permissible to bet at odds in the convex hull of your representor”.
But I’m not aware of an argument that representors should be convex in general.
If there is such an argument, my guess is that the way things would work is: We start with the non-convex set of distributions that seem no less reasonable than each other, and then add in whichever other distributions are needed to make it convex. But there would be no particular reason we’d need to interpret these other distributions as “reasonable” precise beliefs, relative to the distributions in the non-convex set we started with.
And, the kind of precise distribution P that would rationalize e.g. working on shrimp welfare seems to be the analogue of “betting at 0.5” in my example above. That is:
Our actual “set of distributions that seem no less reasonable than each other” would include some distributions that imply large positive long-term EV from working on shrimp welfare, and some that imply large negative long-term EV.
Whereas the distributions like P that imply vanishingly small long-term EV — given any evidence too weak to resolve our cluelessness w.r.t. long-term welfare — would lie in the convex hull. So betting at odds P would be permissible, and yet this wouldn’t imply that P is “reasonable” as precise beliefs.