This is maybe half or more of what Robin Hanson wrote about back when it was still all on overcomingbias.com
Benquo
Karma: 10,252
Link seems broken
The “different masters” thing is a special case of the problem of accepting feedback (i.e. learning from approval/disapproval or reward/punishment) from approval functions in conflict with each other or your goals. Multiple humans trying to do the same or compatible things with you aren’t “different masters” in this sense, since the same logical-decision-theoretic perspective (with some noise) is instantiated on both.
But also, there’s all sorts of gathering data from others’ judgment that doesn’t fit the accountability/commitment paradigm.
Here are a couple specific ways I expect that lumping these cases together will case bad decisions (Raemon’s comment helped me articulate this):
If you don’t notice that some character types are stable under pressure, you’ll overestimate the short-run power of incentives relative to selection effects. You can in fact ask people to act against their “incentives” (example: the Replication Project, whistleblowers like Daniel Ellsberg and Edward Snowden, people who return lost wallets) when conditions permit such people to exist, though this only lasts intergenerationally if longer-run incentives point the other way. What you can’t do is expect people in particular positions where there’s strong selection for certain traits, to exhibit incompatible traits. (Related: Less Competition, More Meritocracy?, No, it’s not The Incentives—it’s you)
If you assume learning and responding to pressure are the same, you’ll put yourself in a position where you’re forced to do things way more than is optimal. In hindsight, instead of going for a master’s degree in math and statistics, I wish I’d (a) thought about what math I was curious about and tried to take the least-effort path to learning it, and (b) if I had motivation problems, examined whether I even wanted to learn the math, instead of outsourcing motivation to the academic system and my desire (more precisely, anxiety) to pass tests. I massively underinvested in playing with or other low-pressure high-intrinsic-motivation exploration of things I was just … interested in. (Related: The Costs of Reliability, The Order of the Soul).
I feel like all of this mixes together info sources and incentives, so it feels a bit wrong to say I agree, but also feels a bit wrong to say I disagree.
I think Critch is basically correct here; it makes more sense to model distractions or stress due to internal conflict as accumulating in some contexts, rather than willpower as a single quantity being depleted.
It seems like you’re imagining a context that isn’t particularly conducive to making intellectual progress. Otherwise, why would it be the case that John feels the need to regularly argue for veganism? If it’s not obvious to the others that John’s not worth engaging with, they should double-crux and be done with it. The “needs” framing feels like a tell that talking, in this context, is mainly about showing that you have broadcast rights, rather than about informing others.
The main case I can imagine where a truth-tracking group should be rationing attention like this, is an emergency where there’s a time-sensitive question that needs to be answered, and things without an immediate bearing on it need to be suppressed for the duration.
My steelman of this position is something like, “I favored focusing instrumental rationality because it seemed, well, useful. At the time I figured that this was just a different subject than epistemic rationality, & focusing on it would at worst mean less progress improving the accuracy of our beliefs. But in hindsight this involved allowing epistemics to get worse for the sake of more instrumental success. I’ve now updated towards that having been a bad tradeoff.”
How close is that?
I don’t think I find that objectionable, it didn’t seem particularly interesting as a claim. It’s as old as “you can only serve one master,” god vs mammon, etc etc—you can’t do well at accountability to mutually incompatible standards. I think it depends a lot on the type and scope of accountability, though.
If the takeaway were what mattered about the post, why include all the other stuff?
Interestingly the readiest example I have at hand comes from Zack Davis. Over email, he suggested four sample edits to Drowning Children are Rare, claiming that this would say approximately the same thing with a much gentler tone. He suggested changing this:
Either charities like the Gates Foundation and Good Ventures are hoarding money at the price of millions of preventable deaths, or the low cost-per-life-saved numbers are wildly exaggerated. My former employer GiveWell in particular stands out as a problem here, since it publishes such cost-per-life-saved numbers, and yet recommended to Good Ventures that it not fully fund GiveWell’s top charities; they were worried that this would be an unfair way to save lives. Either scenario clearly implies that these estimates are severely distorted [...]
To this:
Either charities like the Gates Foundation and Good Ventures are accumulating funds that could be used to prevent millions of deaths, or the low cost-per-life-saved numbers are significantly overestimated. My former employer GiveWell in particular is notable here, since it publishes such cost-per-life-saved numbers, and yet recommended to Good Ventures that it not fully fund GiveWell’s top charities; they were worried about “crowding out” other donors. Either scenario clearly implies that these estimates are systematically mistaken [...]
Some of these changes seemed fine to me (and subsequent edits reflect this), but one of them really does leave out quite a lot, and that kind of suggestion seems pretty typical of the kind of pressure I’m perceiving. I wonder if you can tell which one I mean and how you’d characterize the difference. If not, I’m happy to try explaining, but I figure I should at least check whether the inferential gap here is smaller than I though.
I don’t think I can honestly accept mere explicit endorsements of a high-level opinion here, because as far as I can tell, such endorsements are often accompanied by behavior, or other endorsements, that seem (to me) to contradict them. I guess that’s why the examples have attracted so much attention—I have more of an expectation that they correspond to the intuitions people will make decisions with, and those are what I want to be arguing with.
Having written that, it occurs to me that I too could do a better job giving examples that illustrate the core considerations I’m trying to draw attention to, so I’ll make an effort to do that in the future.
It might help for me to also try to make a positive statement of what I think is at stake here.
I agree with the underlying point that side-channel communication around things like approval is real and common, and it’s important to be able to track and criticize such communication.
What I see as under threat is the ability to say in a way that’s actually heard, not only that opinion X is false, but that the process generating opinion X is untrustworthy, and perhaps actively optimizing in an objectionable direction. Frequently, attempts to say this are construed primarily as moves to attack some person or institution, pushing them into the outgroup. Frequently, people suggest to me an “equivalent” wording with a softer tone, which in fact omits important substantive criticisms I mean to make, while claiming to understand what’s at issue.
My attention has been on which parts of speech it is legitimate to call out.
Do you think anyone in this conversation has an opinion on this beyond “literally any kind of speech is legitimate to call out as objectionable, when it is in fact objectionable”? If so, what?
I thought we were arguing about which speech is in fact objectionable, not which speech it’s okay to evaluate as potentially objectionable. If you meant only to talk about the latter, that would explain how we’ve been talking past each other.
It feels like you keep repeating the 101 arguments and I want to say “I get them, I really get them, you’re boring me”—can you instead engage with why I think we can’t use “but I’m saying true things” as free license to say anything in way whatsoever? That this doesn’t get you a space where people discuss truth freely.
I think some of the problem here is that important parts of the way you framed this stuff seemed as though you really didn’t get it—by the Gricean maxim of relevance—even if you verbally affirmed it. Your framing didn’t distinguish between “don’t say things through the side channels of your speech” and “don’t criticize other participants.” You provided a set of examples that skipped over the only difficult case entirely. The only example you gave of criticizing the motives of a potential party to the conversation was gratuitous insults.
(The conversational move I want to recommend to you here is something like, “You keep saying X. It sort of seems like you think that I believe not-X. I’d rather you directly characterized what you think I’m getting wrong, and why, instead of arguing on the assumption that I believe something silly.” If you don’t explicitly invite this, people are going to be inhibited about claiming that you believe something silly, and arguing to you that you believe it, since it’s generally rude to “put words in other people’s mouths” and people get unhelpfully defensive about that pretty reliably, so it’s natural to try to let you save face by skipping over the unpleasantness there.)
I think there’s also a big disagreement about how frequently someone’s motivations are interfering with their ability to get the right answer, or how frequently we should bring up something like this. It seems like you’re thinking of that as something like the “nuclear option,” which will of course be a self-fulfilling prophecy, but also prevents anything like a rationality forum from working, given how much bias comes from trying to get the wrong answer.
Then there’s also a problem where it’s a huge amount of additional work to separate out side channel content into explicit content reliably. Your response to Zack’s “What? Why?” seemed to imply that it was contentless aggression it would be costless to remove. It was in fact combative, and an explicit formulation would have been better, but it’s a lot of extra work to turn that sort of tone into content reliably, and most people—including most people on this forum—don’t know how to do it. It’s fine to ask for extra work, but it’s objectionable to do so while either implying that this is a free action, or ignoring the asymmetric burdens such requests impose.
I think this is missing one of the most important benefits of things like double crux: the potential for strong updates outside the domain of the initial agreement, for both parties. See also Benito’s A Sketch of Good Communication.
I mean the section titled “2. Secondary information perceived in your message is upsetting.”
“Double crux is for building products” is true mostly because of the more general fact that epistemic rationality is for shared production relationships.
I didn’t think I was disagreeing with you—I meant to refer to the process of publicly explicitly awarding points to offset the implied reputational damage
But much better would be that pointing out that someone was in fact doing harm would not be seen as punishment, if they stop when this is pointed out. In the world in which doing things is appreciated and rewarded, saying “I see you trying to do a thing! I think it’s harmful and you should stop.” and you saying “oops!” should net you points without me having to say “POINTS!”
Huh. I think part of what’s bothering me here is that I’m reading requests to award points (on the assumption that otherwise people will assign credit perversely) as declaring intent to punish me if I publicly change my mind in a way that’s not savvy to this game, insofar as implying that perverse norms are an unchangeable fait accompli strengthens those norms.
I think this schema could benefit from a distinction between rules for internal and external consumption. For external consumption there’s some benefit to having (implied) policies for exceptions that are responsive to likely costs, both the internal costs of acting according to the rule in an emergency, and the external cost of having one’s expectations violated.
But for internal consumption, it makes more sense to, as Said Achmiz points out, just change the rule to a better one that gets the right answer in this case (and all the prior ones). I think people are confused by this in large part because they learned, from authoritarian systems, to rule themselves as though they were setting rules for external consumption or accountability, instead of reasoning about and doing what they want.
This leads to a weird tension where the same person is sternly setting rules for themselves (and threatening to shame themselves for rule violations), and trying to wiggle out of those same rules as though they were demands by a malevolent authority.