Just, the sort of thing that you should say ‘ah, that makes sense. I will work on that’ for the future.
It’s actually not clear to me that I should work on that. As a professional hazard of my other career, I’m pretty used to people trying to use “You would be more persuasive if you were nicer” as an attempted silencing tactic; if I just believed everyone who told me that, I would never get anything done.
What is your incredulity here aiming to accomplish?
I genuinely feel incredulous and am trying to express what I’m actually thinking in clear language? I mean, it’s also totally going to be the case that the underlying generator of “genuinely felt incredulity” is no doubt going to be some sort of subconscious monkey-politics status move designed by evolution to make myself look good at the expense of others. It’s important to notice that! But the mere fact of having noticed that doesn’t make the feeling go away, and given that the feeling is there, it’s probably going to leak into my writing. I could expend more effort doing a complicated System-2 political calculation that tries to simulate you and strategically compute what words I should say in order to have the desired effect on you. But not only is that more work than saying what I’m actually thinking in clear language, I also expect it to result in worse writing. Use the native architecture!
I mean, if it’ll help, we can construct a narrative in which my emotion of incredulity that was designed by evolution to make me look good, actually makes me look bad in local social reality? That’s a win-win Pareto improvement: I don’t have to mutilate my natural writing style in the name of so-called “cooperative” norms, and you don’t have to let my monkey-politics brain get away with “winning” the interaction.
How about this? Incredulity is, definitionally, a failed prediction. The fact that I felt incredulous means that my monkey status instincts are systematically distorting my anticipations about the world, making me delusionally perceive things as “obvious” exactly when they’re things that I coincidentally happened to already know, and not because of their actual degree-of-obviousness as operationalized by what fraction of others know them. (And conversely, I’ll delusionally perceive things as “nonobvious” exactly when I coincidentally happened to not-know them.)
(Slaps forehead) Hello, Megan! Ten years into this “rationality” business, and here I am still making rookie mistakes like this! How dumb can I get?
I think you should prioritize learning to simulate other minds a bit
Thanks, this is a good suggestion! I probably am below average at avoiding the typical mind fallacy. You should totally feel superior to me on this account!
Okay, but I thought the idea was that instrumental rationality and epistemic rationality are very closely related. Two sides of the same coin, not two flavors of good thing that sometimes trade off against each other. That agents achieve their goals by means of building accurate models, and using those models to “search out paths through probability” that steer the world into the desired goal-state. If the models aren’t accurate, the instrumental probability-bending magic doesn’t work and cannot work.
I definitely agree that there could exist perverse situations where there are instrumental tradeoffs to be made in truthseeking of the kind I and others have been suspicious of. For lack of a better term, let me call these “instrumentally epistemic” arguments: claims of the form, “X is true, but the consequences of saying it will actually result in less knowledge on net.” I can totally believe that some instrumentally epistemic arguments might hold. There’s nothing in my understanding of how the universe works that would prevent that kind of scenario from happening.
But in practice, with humans, I expect that a solid supermajority of real-world attempts to explicitly advocate for norm changes on “instrumentally epistemic” grounds are going to be utterly facile rationalizations with the (typically unconscious) motivation of justifying cowardice, intellectual dishonesty, ego-protection, &c.
I (somewhat apologetically) made an “instrumentally epistemic” argument in a private email thread recently, and Ben seemed super pissed in his reply (bold italics, incredulous tone, ”?!?!?!?!?!” punctuation). But the thing is—even if I might conceivably go on to defend a modified form of my original argument—I can’t blame Ben for using a pissed-off tone in his reply. “Instrumentally epistemic” arguments are an enormous red flag—an infrared flag thirty meters wide. Your prior should be that someone making an “instrumentally epistemic” argument can be usefully modeled as trying to undermine your perception of reality and metaphorically slash your tires (even if their conscious phonological loop never contains the explicit sentence “And now I’m going to try to undermine Ray Arnold’s perception of reality”).
Now, maybe that prior can be overcome for some arguments and some arguers! But the apparent failure of the one making the “instrumentally epistemic” argument to notice the thirty-meter red flag, is another red flag.
I don’t think the hospital example does the situation justice. The trade-off of choosing whether to spend money on a heart transplant or nurse salaries doesn’t seem analogous to choosing between truth and the occasional allegedly-instrumentally-epistemic lie (like reassuring your interlocutor that you respect them even when you don’t, in fact, respect them). Rather, it seems more closely analogous to choice of inquiry area (like whether to study truths about chemistry, or truths about biology), with “minutes of study time” as the resource to be allocated rather than dollars.
If we want a maximally charitable medical analogy for “instrumentally epistemic” lies, I would instead nominate chemotherapy, where we deliberately poison patients in the hope of hurting cancer cells more than healthy cells. Chemotherapy can be good if there’s solid evidence that you have a specific type of cancer that responds well to that specific type of chemotherapy. But you should probably check that people aren’t just trying to poison you!
Over the past couple years, I have updated to “yes, LessWrong should be the place focused on truthseeking.”
Updated to? This wording surprises me, because I’m having trouble forming a hypothesis as to what your earlier position could have been. (I’m afraid I haven’t studied your blogging corpus.) What else is this website for, exactly?
I got the impression that statements of the sort “yay truth as the only sacred value” received strong support; personally I find that off-putting in many contexts.
I also find it off-putting in many contexts—perhaps most contexts. But if there’s any consequentialist value in having one space in the entire world where (within the confines of that space) truth is the only sacred value, perhaps lesswrong.com is a Schelling point?
The decision procedure IS the values [...] taught through examples and rituals and anecdotes and example and the weights on the neural nets in people’s heads
That makes sense; I agree that culture (which is very complicated and hard to pin down) is a very important determinant of outcomes in organizations. One thing that’s probably important to study (that I wish I understood better) is how subcultures develop over time: as people leave and exit the organization over time, the values initially trained into the neural net may drift substantially.
or they’re doing some sort of socratic move (in the latter case, this is a style of conversation I’d rather not have on my posts
Very well. I will endeavor to be more direct.
there are clear answers to them if you spend a few minutes steelmanning how the aforementioned organization would work well
The fourth virtue is evenness! If you first write at the bottom of a sheet of paper, “And therefore, the aforementioned organization would work well!”, it doesn’t matter what arguments you write above it afterward—the evidential entanglement between your position and whatever features-of-the-world actually determine organizational success, was fixed the moment you determined your conclusion. After-the-fact steelmanning that selectively searches for arguments supporting that conclusion can’t help you design better organizations unless they have the power to change the conclusion. Yes requires the possibility of no.
they’re looking for impossible certainty in an obviously context specific and highly variable situation
We’re looking for a decision procedure. “It’s context-specific; it depends” is a good start, but a useful proposal needs to say more about what it depends on.
A simple example of a decision procedure might be “direct democracy.” People vote on what to do, and whichever proposal has more votes is implemented. This procedure provides a specific way to proceed when people don’t agree on what to do: they vote!
In both the OP and your response to me, you tell a story about people successfully talking out their differences, but a robust institution needs to be able to function even when people can’t talk it out—and the game theory of “What happens if we can’t talk it out” probably ends up shaping people’s behavior while talking it out.
For example, suspects of a police investigation might be very cooperative with the “good cop” who speaks with a friendly demeanor and offers the suspect a cup of coffee: if you look at the radically transparent video of the interview, you’ll just see two people having a perfectly friendly conversation about where the suspect was at 8:20 p.m. on the night of the seventeenth and whether they have witnesses to support this alibi. But the reason that conversation is so friendly is because the suspect can predict that the good cop’s partner might not be so friendly.
our norm of radical transparency means that this and all similar conversations I have like this will be recorded and shared with everyone, and any such political moves by me will be laughably transparent.
And the decision algorithm that your brain uses to decide who to sit down is also recorded, one imagines? In accordance with our norm of radical transparency.
The general rule is that people should give equal weight to their own needs, the needs of the people they’re interacting with, and the needs of the organization as a whole.
I’m terribly sorry, but I’m afraid I’m having a little bit of trouble working out the details of exactly how this rule would be applied in practice—could you, perhaps, possibly, help me understand?
Suppose Jill comes to Jezebel and says, “Jezebel, by mentioning the hidden Bayesian structure of language and cognition so often, you’re putting your own needs above the needs of those you’re interacting with, and those of the organization as a whole.”
Jezebel says, “Thanks, I really value your opinion! However, I’ve already taken everyone’s needs into account, and I’m very confident I’m already doing the right thing.”
What happens?
10 is vague, and lacks examples.
That’s fair. For a more concrete example, see the immortal Scott Alexander’s recent post “Against Lie Inflation” (itself a reply to discussion with Jessica Taylor on her Less Wrong post “The AI Timelines Scam”). Alexander argues:
The word “lie” is useful because some statements are lies and others aren’t. [...] The rebranding of lying is basically a parasitic process, exploiting the trust we have in a functioning piece of language until it’s lost all meaning[.]
I read Alexander as making essentially the same point as “10.” in the grandparent, with G = “honest reports of unconsciously biased beliefs (about AI timelines)” and H = “lying”.
a) GiveWell does publish cost-effectiveness estimates. I found them in a few clicks. So Ben’s claim is neither dishonest nor false.
While I agree that this is a sufficient rebuttal of Ray’s “dishonest and/or false” charge (Ben said that GiveWell publishes such numbers, and GiveWell does, in fact, publish such numbers), it seems worth acknowleding Ray’s point about context and reduced visibility: it’s not misleading to publish potentially-untrustworthy (but arguably better than nothing) numbers surrounded by appropriate caveats and qualifiers, even when it would be misleading to loudly trumpet the numbers as if they were fully trustworthy.
That said, however, Ray’s “GiveWell goes to great lengths to hide those numbers” claim seems false to me in light of an email I received from GiveWell today (the occasion of my posting this belated comment), which reads, in part:
GiveWell has made a lot of progress since your last recorded gift in 2015. Our current top charities continue to avert deaths and improve lives each day, and are the best giving opportunities we’re aware of today. To illustrate, right now we estimate that for every $2,400 donated to Malaria Consortium for its seasonal malaria chemoprevention program, the death of a child will be averted.
(Bolding mine.)
In order to combat publication bias, I should probably tell the Open Thread about a post idea that I started drafting tonight but can’t finish because it looks like my idea was wrong. Working title: “Information Theory Against Politeness.” I had drafted this much—
Suppose the Quality of a Blog Post is an integer between 0 and 15 inclusive, and furthermore that the Quality of Posts is uniformly distributed. Commenters can roughly assess the Quality of a Post (with some error in either direction) and express their assessment in the form of a Comment, which is also an integer between 0 and 15 inclusive. If the True Quality of a post is , then the assessment expressed in a Comment on that Post follows the probability distribution
(Notice the “wraparound” between 15 and 0: it can be hard for a humble Commenter to tell the difference between brilliance-beyond-their-ken, and utter madness!)
The entropy of the Quality distribution is = 4 bits: in order to inform someone about the Quality of a Post, you need to transmit 4 bits of information. Comments can be thought of as a noisy “channel” conveying information about the post.
The mutual information between a Comment, and the Post’s Quality, is equal to the entropy of the distribution of Comments (which is going to be 4 bits, by symmetry), minus the entropy of a Comment given the Post’s Quality (which is ≈ 1.58). So the “capacity” of a single Comment is around 4 − 1.58 = 2.42 bits. On average, in expectation across the multiverse, &c., we only need to read 4⁄2.42 ≈ 1.65 Comments in order to determine the Quality of a Post. Efficient!
Now suppose the Moderators introduce a new Rule: it turns out Comments below 10 are Rude and hurt Post Authors’ Feelings. Henceforth, all Comments must be an integer between 10 and 15 inclusive, rather than between 0 and 15 inclusive!
… and then I was expecting that the restricted range imposed by the new Rule would decrease the expressive power of Comments (as measured by mutual information), but now I don’t think this is right: the mutual information is about the noise in Commenter’s perceptions, not the “coarseness” of the “buckets” in which it is expressed: lg(16) − lg(3) has the same value as lg(8) − lg(1.5).
obvious
Yeah, I would have expected Jessica to get it, except that I suspect she’s also executing a strategy of habitual Socratic irony (but without my additional innovation of immediately backing down and unpacking the intent when challenged), which doesn’t work when both sides of a conversation are doing it.
You caught me—introspecting, I think the grandparent was written in a spirit of semi-deliberate irony. (“Semi” because it just felt like the “right” thing to say there; I don’t think I put a lot of effort into modeling how various readers would interpret it.)
Roland is speculating that the real reason for intentionally incomplete explanations in the handbook is different from the stated reason, and I offered a particularly blunt phrasing (“we don’t want to undercut our core product”) of the hypothesized true reason, and suggested that that’s what the handbook would have said in that case. I think I anticipated that a lot of readers would find my proposal intuitively preposterous: “everyone knows” that no one would matter-of-factly report such a self-interested rationale (especially when writing on behalf of an organization, rather than admitting a vice among friends). That’s why the earlier scenes in the 2009 film The Invention of Lying, or your post “Act of Charity”, are (typically) experienced as absurdist comedy rather than an inspiring and heartwarming portrayal of a more truthful world.
But it shouldn’t be absurd for the stated reason and the real reason to be the same! Particularly for an organization like CfAR which is specifically about advancing the art of rationality. And, I don’t know—I think sometimes I talk in a way that makes me seem more politically naïve than I actually am, because I feel as if the “naïve” attitude is in some way normative? (“You really think someone would do that? Just go on the internet and tell lies?”) Arguably this is somewhat ironic (being deceptive about your ability to detect deception is probably not actually the same thing as honesty), but I haven’t heretofore analyzed this behavioral pattern of mine in enough detail to potentially decide to stop doing it??
I think another factor might be that I feel guilty about being “mean” to CfAR in the great-great-great grandparent comment? (CfAR isn’t a person and doesn’t have feelings, but my friend who works there is and does.) Such that maybe the emotional need to signal that I’m still fundamentally loyal to the “mainstream rationality” tribe (despite the underlying background situation where I’ve been collaborating with you and Ben and Michael to discuss what you see as fatal deficits of integrity in “the community” as presently organized) interacted with my preëxisting tendency towards semi-performative naiveté in a way that resulted in me writing a bad blog comment? It’s a good thing you were here to hold me to account for it!
I suspect that this is the real reason.
It’s pretty uncharitable of you to just accuse CfAR of lying like that! If the actual reason were “Many of the explanations here are intentionally approximate or incomplete because we predict that this handbook will be leaked and we don’t want to undercut our core product,” then the handbook would have just said that.
There are many subjects where written instructions are much less valuable than instruction that includes direct practice: circling, karate, meditation, dancing, etc.
Yes, I agree: for these subjects, the “there’s a lot of stuff we don’t know how to teach in writing” disclaimer I suggested in the grandparent would be a big understatement.
a syllabus is useless (possibly harmful) for teaching economics to people who have bad assumptions about what kind of questions economics answers
Useless, I can believe. (The extreme limiting case of “there’s a lot of stuff we don’t know how to teach in this format” is “there is literally nothing we know how to teach in this format.”) But harmful? How? Won’t the unexpected syllabus section titles at least disabuse them of their bad assumptions?
Reading the sequences [...] are unlikely to have much relevance to what CFAR teaches.
Really? The tagline on the website says, “Developing clear thinking for the sake of humanity’s future.” I guess I’m having trouble imagining a developing-clear-thinking-for-the-sake-of-humanity’s-future curriculum for which the things we write about on this website would be irrelevant. The “comfort zone expansion” exercises I’ve heard about would qualify, but Sequences-knowledge seems totally relevant to something like, say, double crux.
(It’s actually pretty weird/surprising that I’ve never personally been to a CfAR workshop! I think I’ve been assuming that my entire social world has already been so anchored on the so-called “rationalist” community for so long, that the workshop proper would be superfluous.)
I like the no-nonsense section titles!
I also like the attempt to graphically teach the conflict between the different fairness desiderata using squares, but I think I would need a few more intermediate diagrams (or probably, to work them out myself) to really “get it.” I think the standard citation here is “Inherent Trade-Offs in the Fair Determination of Risk Scores”, but that presentation has a lot more equations and fewer squares.
a harder time grasping a given technique if they’ve already anchored themselves on an incomplete understanding
This is certainly theoretically possible, but I’m very suspicious of it on reversal test grounds: if additional prior reading is bad, then why isn’t less prior reading even better? Should aspiring rationalists not read the Sequences for fear of an incomplete understanding spoiling themselves for some future $3,900 CfAR workshop? (And is it bad that I know about the reversal test without having attended a CfAR workshop?)
I feel the same way about schoolteachers who discourage their students from studying textbooks on their own (because they “should” be learning that material by enrolling in the appropriate school course). Yes, when trying to learn from a book, there is some risk of making mistakes that you wouldn’t make with the help of a sufficiently attentive personal tutor (which, realistically, you’re not going to get from attending lecture classes in school anyway). But given the alternative of placing my intellectual trajectory at the mercy of an institution that has no particular reason to care about my welfare, I think I’ll take my chances.
Note that I’m specifically reacting to the suggestion that people not read things for their own alleged benefit. If the handbook had just said, “Fair warning, this isn’t a substitute for the workshop because there’s a lot of stuff we don’t know how to teach in writing,” then fine; that seems probably true. What I’m skeptical of is hypothesized non-monotonicity whereby additional lower-quality study allegedly damages later higher-quality study. First, because I just don’t think it’s true on the merits: I falsifiably predict that, e.g., math students who read the course textbook on their own beforehand will do much better in the course than controls who haven’t. (Although the pre-readers might annoy teachers whose jobs are easier if everyone in the class is obedient and equally ignorant.) And second, because the general cognitive strategy of waiting for the designated teacher to spoonfeed you the “correct” version carries massive opportunity costs when iterated (even if spoonfeeding is generally higher-quality than autodidactism, and could be much higher-quality in some specific cases).
As it happens, the case of speeding also came up in the comments on the OP. Yarkoni writes:
[...] I think the point I’m making actually works well for speeding too: when you get pulled over by a police officer for going 10 over the limit, nobody is going to take you seriously if your objection to the ticket is “but I’m incentivized to go 10 over, because I can get home a little faster, and hardly anyone ever gets pulled over at that speed!” The way we all think about speeding tickets is that, sure, there may be reasons we choose to break the law, but it’s still our informed decision to do so. We don’t try shirk the responsibility for speeding by pretending that we’re helpless in the face of the huge incentive to get where we’re going just a little bit faster than the law actually allows. I think if we looked at research practice the same way, that would be a considerable improvement.
Thanks, I appreciate it a lot! You should be questioning my “side” as harshly as you see fit, because if you ask questions I can’t satisfactorily answer, then maybe my side is wrong, and I should be informed of this in order to become less wrong.
The mechanism by which saying true things leads to more knowledge is at least straightforward: you present arguments and evidence, and other people evaluate those arguments and evidence using the same general rules of reasoning that they use for everything else, and hopefully they learn stuff.
In order for saying true things to lead to less knowledge, we need to postulate some more complicated failure mode where some side-effect of speech disrupts the ordinary process of learning. I can totally believe that such failure modes exist, and even that they’re common. But lately I seem to be seeing a lot of arguments of the form, “Ah, but we need to coordinate in order to create norms that make everyone feel Safe, and only then can we seek truth.” And I just … really have trouble taking this seriously as a good faith argument rather than an attempt to collude to protect everyone’s feelings? Like, telling the truth is not a coordination problem? You can just unilaterally tell the truth.
Hm, I think there’s a risk of signal miscalibration here. Just because I feel exasperated and this emotion leaks into my writing, doesn’t necessarily mean implied probabilities close to 1? (Related: Say It Loud. See also my speculative just-so story about why the incredulity is probably non-normative.)
(It’s 1:20 a.m. on Sunday and I’ve used up my internet quota for the weekend, so it might take me a few days to respond to future comments.)