Thanks, this was thought-provoking. I feel confused about how action-relevant this idea is, though.
For one, let’s grant that (a) “researching considerations + basing my recommendation on the direction of the considerations” > (b) “researching considerations + giving no recommendation”. This doesn’t tell me how to compare (a) “researching considerations + basing my recommendation on the direction of the considerations” vs. (c) “not doing research”. Realistically, the act of “doing research” would have various messy effects relative to, say, doing some neartermist thing — so I’d think (a) is incomparable with (c). (More on this here.)
But based on the end of your comment, IIUC you’re conjecturing that we can compare plans based on a similar idea to your example even if no “research” is involved, just passively gaining info. If so:
It seems like this wouldn’t tell me to change anything about what I work on in between times when someone asks for my recommendation.
Suppose I recommend that someone do more of [intervention that I’ve positively updated on]. Again, their act of investing more in that intervention will presumably have lots of messy side effects, besides “more of the intervention gets implemented” in the abstract. So I should only be clueful that this plan is better if I’ve “positively updated” on the all-things-considered set of effects of this person investing more in that intervention. (Intuitively this seems like an especially high bar.)
Huh, yeah, I suppose that side-effects from research could swing it. I’m not really sure how to analyze this.
Here’s a bit of an expanded line of thinking/intuition, following the same thread as I started on in my previous comment, but getting a bit less concrete and a bit more speculative. It’s an argument for why this kind of thing will (maybe) rule out some policies as impermissible:
In order for you to not suffer sure-losses, you must be representable as an EV-maximizing agent, which means that there must be some beliefs for which your entire policy (your mapping from all possible future observations, to all possible actions) is optimal w.r.t. total welfare.
(Of course, as bounded agents, even if we try hard to be EV-maximizing-w.r.t.-precise-credences, we’ll be suffering sure-losses as viewed from the perspective of epistemically superior agents. Some of our beliefs are contradictory, etc. So what we really want here is some milder version of this criterion that admits to the limitation of bounded agents, but still condemns inefficiencies that we are capable of avoiding.)
An intuitively compelling criterion is: these precise beliefs (which you are representable as holding) are within the bounds of your imprecise credences.
I’m not sure if this criterion follows naturally from dynamic strong maximality, or if it’s a plausible “extra criterion” to add on top, or if you think it’s a bad criterion that we shouldn’t assume.
For the rest of this comment, I’ll assume it.
But when I picture myself living a life under cluelessness about impartial long-run welfare, as I currently intuitively understand such lives to look like (doing stuff like “working on animal welfare”, or “caring for friends and family”, or “researching/advocating for cluelessness”), and in particular, when I picture the big complex object that is the policy of a person doing these sorts of actions — including all the ways they could react to all possible information — it’s really unclear whether even severly imprecise credences should reasonably include any set of beliefs under which that policy is optimal w.r.t. total welfare.
Intuitively, it seems like that might require oddly contorted beliefs. For example, for some reason, your actions change hugely when you learn about details of slaughter methods for shrimp, but change not at all when you learn what-seems-like-it-ought-to-be highly relevant information about the long-run consequences of your actions. Or perhaps your actions change a lot when you discover a new consideration for how to do reasoning under cluelessness. Are there really any “reasonable” precise beliefs (i.e. within the bounds of your imprecise credences) that prescribe that precise policy?
(One possible direction to explore: What’s the “size” of “the set of all reasonable precise credences”, and what’s the “size” of “the set of all possible policies”. If the former is vastly larger than the latter, then maybe it doesn’t seem so unlikely that there will always be some set of “reasonable credences” that recommends any particular policy. But if not, then constraining yourself to only follow policies that are recommended by some possible “reasonable credences” will probably significantly limit the set of policies that are permissible to choose.)
An intuitively compelling criterion is: these precise beliefs (which you are representable as holding) are within the bounds of your imprecise credences.
I think this is the step I reject. By hypothesis, I don’t think the coherence arguments show that the precise distribution P that I can be represented as optimizing w.r.t. corresponds to (reasonable) beliefs. P is nothing more than a mathematical device for representing some structure of behavior. So I’m not sure why I should require that my representor — i.e., the set of probability distributions that would be no less reasonable than each other if adopted as beliefs[1] — contains P.
I think I maybe figured out how to show that P must be in the representor.
You ought to assign non-0 probability that you’ll be asked to bet on arbitrary questions. In order to not have your policy be dominated, dynamic maximality will require that you commit in advance to the odds that you’d be on (after seeing arbitrary evidence). Clearly you should be on odds P. And it’s only permissible to bet at odds that are inside your representor.
(Now strictly speaking, there are some nuances about what kind of questions you can be convinced that you’ll be betting on, given that some of them might be quite hard to measure/verify even post-hoc. But since we’re just talking about non-0 probability of being convinced that you’re really betting on a question, I don’t think this should be too restrictive. And even non-”pure” bets, that only indirectly gets at some question q, will contribute to forcing P’s belief in q inside of your representor, I think.)
Sorry, I don’t understand the argument yet. Why is it clear that I should bet on odds P, e.g., if P is the distribution that the CCT says I should be represented by?
Because you couldn’t be represented as being an EV-maximizer with beliefs P if you were betting using some odds other than P. Because that would lead to lower expected value. (Assuming that pay-offs are going to be proportional to some proper scoring rule.)
“It’s only permissible to bet at odds that are inside your representor” is only true if the representor is convex. If my credence in some proposition X is, say, P(X) = (0.2, 0.49) U (0.51, 0.7), IIUC it’s permissible to bet at 0.5. I guess the claim that’s true is “It’s only permissible to bet at odds in the convex hull of your representor”.
But I’m not aware of an argument that representors should be convex in general.
If there is such an argument, my guess is that the way things would work is: We start with the non-convex set of distributions that seem no less reasonable than each other, and then add in whichever other distributions are needed to make it convex. But there would be no particular reason we’d need to interpret these other distributions as “reasonable” precise beliefs, relative to the distributions in the non-convex set we started with.
And, the kind of precise distribution P that would rationalize e.g. working on shrimp welfare seems to be the analogue of “betting at 0.5” in my example above. That is:
Our actual “set of distributions that seem no less reasonable than each other” would include some distributions that imply large positive long-term EV from working on shrimp welfare, and some that imply large negative long-term EV.
Whereas the distributions like P that imply vanishingly small long-term EV — given any evidence too weak to resolve our cluelessness w.r.t. long-term welfare — would lie in the convex hull. So betting at odds P would be permissible, and yet this wouldn’t imply that P is “reasonable” as precise beliefs.
Little bit late, but I’ve been reading about cluelessness and I think Lukas’ example is a very good intuition pump against radical cluelessness and this counterpoint doesn’t seem to work. The discussion moved elsewhere afterwards but this example strikes me as strong. In particular, I don’t think it matters whether research typically has the side effect in practice, because it is plausible (at least in theory) to have a world where it doesn’t.
To see this, take Lukas’ above example except assume that “research” will be done without a side effect. For example, “research” could mean taking 5 seconds to look up the recommendation of a newly released report. This report analyzes new information you did not previously know about and summarizes into a binary positive or a negative consideration. Assume that the new information this report gives is somewhat valuable, but you will continue to be radically clueless about alignment research after reading it. Also assume that any knock-on effects from the 5 seconds taken to look up this report cancel out (simple cluelessness) so b) and c) are equivalent (not just incomparable).
Worlds a) and b) and worlds a) and c) are incomparable from one standpoint because you are still radically clueless about alignment research being good. But from another standpoint should cluefully say that a > (b = c) because it is good to move $1000 from worlds where you get a negative update for alignment research to worlds where you get a positive update for alignment research. Due to this contradiction, I’m inclined to question the theoretical foundations of the story of imprecise intervals and radical incomparability.
Thanks! To clarify, my counterpoint wasn’t meant to address any objections to incomparability in principle. It was meant to respond to Lukas’s claim that even if you buy the arguments for incomparability in principle and for severe imprecision, some strategies are still comparable.
Anyway, to your point:
Worlds a) and b) and worlds a) and c) are incomparable from one standpoint because you are still radically clueless about alignment research being good.
Could you spell out this claim more? What does it mean for one pair of strategies ((a) and (b)) to be incomparable to another ((a) and (c))? It sounds like you want to say “from one standpoint, (b) and (c) are incomparable”, but I’m not sure.
Thanks for the reply! I think you’re right to push back on Lukas’ point about incomparability in practice. But I (obviously) think there’s a question to be had about imprecise intervals in theory.
My wording was a bit confusing, but I meant to say that (a) is incomparable to (b) — and that a) is incomparable to (c) from one standpoint. In my formulation of the problem, (b) is exactly equal to (c) by simple cluelessness (unlikely in practice but plausible in theory).
I thought of two ways you can try to hold on to complex cluelessness here and why they both seem to struggle. How would you respond?
Would you bite the bullet here and say that (a) and (c) (or, equivalently, (a) and (b)) are incomparable?
If so, we’re clueless about whether we should redirect alignment money when you have a negative update to worlds with a positive update — which I find very unintuitive.
If you try bracketing away this cluelessness, then I’m also skeptical.
It seems arbitrary whether to bracket before the update on alignment research or after the update, since there’s complex cluelessness in both cases.
Thanks for clarifying! Back to your first comment then:
Worlds a) and b) and worlds a) and c) are incomparable from one standpoint because you are still radically clueless about alignment research being good
I’m still not sure I understand, sorry if I’m missing something basic. Let:
(a*) = “recommend donating to alignment research [without doing research about the sign of alignment beforehand]”
(b*) = “recommend not donating [without doing research about the sign of alignment beforehand]”
As you say, Lukas’s argument implies (a) > (b). (Independently of whether (b) ~ (c).) This holds even if (a*) is incomparable with (b*) (that’s precisely Lukas’s point). I don’t see how “you are still radically clueless about alignment research being good” — i.e., (a*) is incomparable with (b*) — tells me that (a) is incomparable with (b) (or that (a) incomp (c)).
Oops yeah, I believe you’re right. I got confused and I thought we had specified a) incomp b) but in reality we had only specified alignment research after a positive update is incomparable to b*).
Thanks, this was thought-provoking. I feel confused about how action-relevant this idea is, though.
For one, let’s grant that (a) “researching considerations + basing my recommendation on the direction of the considerations” > (b) “researching considerations + giving no recommendation”. This doesn’t tell me how to compare (a) “researching considerations + basing my recommendation on the direction of the considerations” vs. (c) “not doing research”. Realistically, the act of “doing research” would have various messy effects relative to, say, doing some neartermist thing — so I’d think (a) is incomparable with (c). (More on this here.)
But based on the end of your comment, IIUC you’re conjecturing that we can compare plans based on a similar idea to your example even if no “research” is involved, just passively gaining info. If so:
It seems like this wouldn’t tell me to change anything about what I work on in between times when someone asks for my recommendation.
Suppose I recommend that someone do more of [intervention that I’ve positively updated on]. Again, their act of investing more in that intervention will presumably have lots of messy side effects, besides “more of the intervention gets implemented” in the abstract. So I should only be clueful that this plan is better if I’ve “positively updated” on the all-things-considered set of effects of this person investing more in that intervention. (Intuitively this seems like an especially high bar.)
Huh, yeah, I suppose that side-effects from research could swing it. I’m not really sure how to analyze this.
Here’s a bit of an expanded line of thinking/intuition, following the same thread as I started on in my previous comment, but getting a bit less concrete and a bit more speculative. It’s an argument for why this kind of thing will (maybe) rule out some policies as impermissible:
In order for you to not suffer sure-losses, you must be representable as an EV-maximizing agent, which means that there must be some beliefs for which your entire policy (your mapping from all possible future observations, to all possible actions) is optimal w.r.t. total welfare.
(Of course, as bounded agents, even if we try hard to be EV-maximizing-w.r.t.-precise-credences, we’ll be suffering sure-losses as viewed from the perspective of epistemically superior agents. Some of our beliefs are contradictory, etc. So what we really want here is some milder version of this criterion that admits to the limitation of bounded agents, but still condemns inefficiencies that we are capable of avoiding.)
An intuitively compelling criterion is: these precise beliefs (which you are representable as holding) are within the bounds of your imprecise credences.
I’m not sure if this criterion follows naturally from dynamic strong maximality, or if it’s a plausible “extra criterion” to add on top, or if you think it’s a bad criterion that we shouldn’t assume.
For the rest of this comment, I’ll assume it.
But when I picture myself living a life under cluelessness about impartial long-run welfare, as I currently intuitively understand such lives to look like (doing stuff like “working on animal welfare”, or “caring for friends and family”, or “researching/advocating for cluelessness”), and in particular, when I picture the big complex object that is the policy of a person doing these sorts of actions — including all the ways they could react to all possible information — it’s really unclear whether even severly imprecise credences should reasonably include any set of beliefs under which that policy is optimal w.r.t. total welfare.
Intuitively, it seems like that might require oddly contorted beliefs. For example, for some reason, your actions change hugely when you learn about details of slaughter methods for shrimp, but change not at all when you learn what-seems-like-it-ought-to-be highly relevant information about the long-run consequences of your actions. Or perhaps your actions change a lot when you discover a new consideration for how to do reasoning under cluelessness. Are there really any “reasonable” precise beliefs (i.e. within the bounds of your imprecise credences) that prescribe that precise policy?
(One possible direction to explore: What’s the “size” of “the set of all reasonable precise credences”, and what’s the “size” of “the set of all possible policies”. If the former is vastly larger than the latter, then maybe it doesn’t seem so unlikely that there will always be some set of “reasonable credences” that recommends any particular policy. But if not, then constraining yourself to only follow policies that are recommended by some possible “reasonable credences” will probably significantly limit the set of policies that are permissible to choose.)
Thanks for explaining!
I think this is the step I reject. By hypothesis, I don’t think the coherence arguments show that the precise distribution P that I can be represented as optimizing w.r.t. corresponds to (reasonable) beliefs. P is nothing more than a mathematical device for representing some structure of behavior. So I’m not sure why I should require that my representor — i.e., the set of probability distributions that would be no less reasonable than each other if adopted as beliefs[1] — contains P.
I’m not necessarily committed to this interpretation of the representor, but for the purposes of this discussion I think it’s sufficient.
I think I maybe figured out how to show that P must be in the representor.
You ought to assign non-0 probability that you’ll be asked to bet on arbitrary questions. In order to not have your policy be dominated, dynamic maximality will require that you commit in advance to the odds that you’d be on (after seeing arbitrary evidence). Clearly you should be on odds P. And it’s only permissible to bet at odds that are inside your representor.
(Now strictly speaking, there are some nuances about what kind of questions you can be convinced that you’ll be betting on, given that some of them might be quite hard to measure/verify even post-hoc. But since we’re just talking about non-0 probability of being convinced that you’re really betting on a question, I don’t think this should be too restrictive. And even non-”pure” bets, that only indirectly gets at some question q, will contribute to forcing P’s belief in q inside of your representor, I think.)
Sorry, I don’t understand the argument yet. Why is it clear that I should bet on odds P, e.g., if P is the distribution that the CCT says I should be represented by?
Because you couldn’t be represented as being an EV-maximizer with beliefs P if you were betting using some odds other than P. Because that would lead to lower expected value. (Assuming that pay-offs are going to be proportional to some proper scoring rule.)
Oops, right. I think what’s going on is:
“It’s only permissible to bet at odds that are inside your representor” is only true if the representor is convex. If my credence in some proposition X is, say, P(X) = (0.2, 0.49) U (0.51, 0.7), IIUC it’s permissible to bet at 0.5. I guess the claim that’s true is “It’s only permissible to bet at odds in the convex hull of your representor”.
But I’m not aware of an argument that representors should be convex in general.
If there is such an argument, my guess is that the way things would work is: We start with the non-convex set of distributions that seem no less reasonable than each other, and then add in whichever other distributions are needed to make it convex. But there would be no particular reason we’d need to interpret these other distributions as “reasonable” precise beliefs, relative to the distributions in the non-convex set we started with.
And, the kind of precise distribution P that would rationalize e.g. working on shrimp welfare seems to be the analogue of “betting at 0.5” in my example above. That is:
Our actual “set of distributions that seem no less reasonable than each other” would include some distributions that imply large positive long-term EV from working on shrimp welfare, and some that imply large negative long-term EV.
Whereas the distributions like P that imply vanishingly small long-term EV — given any evidence too weak to resolve our cluelessness w.r.t. long-term welfare — would lie in the convex hull. So betting at odds P would be permissible, and yet this wouldn’t imply that P is “reasonable” as precise beliefs.
Little bit late, but I’ve been reading about cluelessness and I think Lukas’ example is a very good intuition pump against radical cluelessness and this counterpoint doesn’t seem to work. The discussion moved elsewhere afterwards but this example strikes me as strong. In particular, I don’t think it matters whether research typically has the side effect in practice, because it is plausible (at least in theory) to have a world where it doesn’t.
To see this, take Lukas’ above example except assume that “research” will be done without a side effect. For example, “research” could mean taking 5 seconds to look up the recommendation of a newly released report. This report analyzes new information you did not previously know about and summarizes into a binary positive or a negative consideration. Assume that the new information this report gives is somewhat valuable, but you will continue to be radically clueless about alignment research after reading it. Also assume that any knock-on effects from the 5 seconds taken to look up this report cancel out (simple cluelessness) so b) and c) are equivalent (not just incomparable).
Worlds a) and b) and worlds a) and c) are incomparable from one standpoint because you are still radically clueless about alignment research being good. But from another standpoint should cluefully say that a > (b = c) because it is good to move $1000 from worlds where you get a negative update for alignment research to worlds where you get a positive update for alignment research. Due to this contradiction, I’m inclined to question the theoretical foundations of the story of imprecise intervals and radical incomparability.
Thanks! To clarify, my counterpoint wasn’t meant to address any objections to incomparability in principle. It was meant to respond to Lukas’s claim that even if you buy the arguments for incomparability in principle and for severe imprecision, some strategies are still comparable.
Anyway, to your point:
Could you spell out this claim more? What does it mean for one pair of strategies ((a) and (b)) to be incomparable to another ((a) and (c))? It sounds like you want to say “from one standpoint, (b) and (c) are incomparable”, but I’m not sure.
Thanks for the reply! I think you’re right to push back on Lukas’ point about incomparability in practice. But I (obviously) think there’s a question to be had about imprecise intervals in theory.
My wording was a bit confusing, but I meant to say that (a) is incomparable to (b) — and that a) is incomparable to (c) from one standpoint. In my formulation of the problem, (b) is exactly equal to (c) by simple cluelessness (unlikely in practice but plausible in theory).
I thought of two ways you can try to hold on to complex cluelessness here and why they both seem to struggle. How would you respond?
Would you bite the bullet here and say that (a) and (c) (or, equivalently, (a) and (b)) are incomparable?
If so, we’re clueless about whether we should redirect alignment money when you have a negative update to worlds with a positive update — which I find very unintuitive.
If you try bracketing away this cluelessness, then I’m also skeptical.
It seems arbitrary whether to bracket before the update on alignment research or after the update, since there’s complex cluelessness in both cases.
Thanks for clarifying! Back to your first comment then:
I’m still not sure I understand, sorry if I’m missing something basic. Let:
(a*) = “recommend donating to alignment research [without doing research about the sign of alignment beforehand]”
(b*) = “recommend not donating [without doing research about the sign of alignment beforehand]”
As you say, Lukas’s argument implies (a) > (b). (Independently of whether (b) ~ (c).) This holds even if (a*) is incomparable with (b*) (that’s precisely Lukas’s point). I don’t see how “you are still radically clueless about alignment research being good” — i.e., (a*) is incomparable with (b*) — tells me that (a) is incomparable with (b) (or that (a) incomp (c)).
Oops yeah, I believe you’re right. I got confused and I thought we had specified a) incomp b) but in reality we had only specified alignment research after a positive update is incomparable to b*).