Someone in the community told me that for me to think AGI probably won’t be developed soon, I must think I’m better at meta-rationality than Eliezer Yudkowsky, a massive claim of my own specialness.
Just zooming in on this, which stood out to me personally as a particular thing I’m really tired of.
If you’re not disagreeing with people about important things then you’re not thinking. There are many options for how to negotiate a significant disagreement with a colleague, including spending lots of time arguing about it, finding a compromise action, or stopping collaborating with the person (if it’s a severe disagreement, which often it can be). But telling someone that by disagreeing they’re claiming to be ‘better’ than another person in some way always feels to me like an attempt to ‘control’ the speech and behavior of the person you’re talking to, and I’m against it.
It happens a lot. I recently overheard someone (who I’d not met before) telling Eliezer Yudkowsky that he’s not allowed to have extreme beliefs about AGI outcomes. I don’t recall the specific claim, just that EY’s probability mass for the claim was in the 95-99% range. The person argued that because EY disagrees with some other thoughtful people on that question, he shouldn’t have such confidence.
(At the time, I noticed I didn’t have to be around or listen to that person and just wandered away. Poor Eliezer stayed and tried to give a thoughtful explanation for why the argument seemed bad.)
Perhaps more important to my subsequent decisions, the AI timelines shortening triggered an acceleration of social dynamics.
I noticed this too. I thought a bunch of people were affected by it in a sort of herd behavior way (not focused so much on MIRI/CFAR, I’m talking more broadly in the rationality/EA communities). I do think key parts of the arguments about how to think about timelines and takeoff are accurate (e.g. 1, 2), but I feel like many people weren’t making decisions because of reasons; instead they noticed their ‘leaders’ were acting scared and then they also acted scared, like a herd.
In both the Leverage situation and the AI timelines situation, I felt like nobody involved was really appreciating how much fuckery the information siloing was going to cause (and did cause) to the way the individuals in the ecosystem made decisions.
This was one of the main motivations behind my choice of example in the opening section of my 3.5 yr old post A Sketch of Good Communication btw (a small thing but still meant to openly disagree with the seeming consensus that timelines determined everything). And then later I wrote about the social dynamics a bunch more 2yrs ago when trying to expand on someone else’s question on the topic.
I affirm the correctness of Ben Pace’s anecdote about what he recently heard someone tell me.
“How dare you think that you’re better at meta-rationality than Eliezer Yudkowsky, do you think you’re special”—is somebody trolling? Have they never read anything I’ve written in my entire life? Do they have no sense, even, of irony? Yeah, sure, it’s harder to be better at some things than me, sure, somebody might be skeptical about that, but then you ask for evidence or say “Good luck proving that to us all eventually!” You don’t be like, “Do you think you’re special?” What kind of bystander-killing argumentative superweapon is that? What else would it prove?
I really don’t know how I could make this any clearer. I wrote a small book whose second half was about not doing exactly this. I am left with a sense that I really went to some lengths to prevent this, I did what society demands of a person plus over 10,000% (most people never write any extended arguments against bad epistemology at all, and society doesn’t hold that against them), I was not subtle. At some point I have to acknowledge that other human beings are their own people and I cannot control everything they do—and I hope that others will also acknowledge that I cannot avert all the wrong thoughts that other people think, even if I try, because I sure did try. A lot. Over many years. Aimed at that specific exact way of thinking. People have their own wills, they are not my puppets, they are still not my puppets even if they have read some blog posts of mine or heard summaries from somebody else who once did; I have put in at least one hundred times the amount of effort that would be required, if any effort were required at all, to wash my hands of this way of thinking.
I’m glad you agree that the behavior Jessica describes is explicitly opposed to the content of the Sequences, and that you clearly care a lot about this. I don’t think anyone can reasonably claim you didn’t try hard to get people to behave better, or could reasonably blame you for the fact that many people persistently try to do the opposite of what you say, in the name of Rationality.
I do think it would be a very good idea for you to investigate why & how the institutions you helped build and are still actively participating in are optimizing against your explicitly stated intentions. Anna’s endorsement of this post seems like reasonably strong confirmation that organizations nominally committed to your agenda are actually opposed to it, unless you’re actually checking. And MIRI/CFAR donors seem to for the most part think that you’re aware of and endorse those orgs’ activities.
When Jessica and another recent MIRI employee asked a few years ago for some of your time to explain why they’d left, your response was:
My guess is that I could talk over Signal voice for 30 minutes or in person for 15 minutes on the 15th, with an upcoming other commitment providing a definite cutoff point and also meaning that it wouldn’t otherwise be an uninterrupted day. That’s not enough time to persuade each other things, but I suspect that neither would be a week, and hopefully it’s enough that you can convey to me any information you want me to know and don’t want to write down. Again, for framing, this is a sort of thing I basically don’t do anymore due to stamina limitations—Nate talks to people, I talk to Nate.
You responded a little bit by email, but didn’t seem very interested in what was going on (you didn’t ask the kinds of followup questions that would establish common knowledge about agreement or disagreement), so your interlocutors didn’t perceive a real opportunity to inform you of these dynamics at that time.
Anna’s endorsement of this post seems like reasonably strong confirmation that organizations nominally committed to your agenda are actually opposed to it,
Presumably Eliezer’s agenda is much broader than “make sure nobody tries to socially enforce deferral to high-status figures in an ungrounded way” though I do think this is part of his goals.
The above seems to me like it tries to equivocate between “this is confirmation that at least some people don’t act in full agreement with your agenda, despite being nominally committed to it” and “this is confirmation that people are actively working against your agenda”. These two really don’t strike me as the same, and I really don’t like how this comment seems like it tries to equivocate between the two.
Of course, the claim that some chunk of the community/organizations Eliezer created are working actively against some agenda that Eliezer tried to set for them is plausible. But calling the above a “strong confirmation” of this fact strikes me as a very substantial stretch.
It’s explicitly opposition to core Sequences content, which Eliezer felt was important enough to write a whole additional philosophical dialogue about after the main Sequences were done. Eliezer’s response when informed about it was:
is somebody trolling? Have they never read anything I’ve written in my entire life? Do they have no sense, even, of irony?
That doesn’t seem like Eliezer agrees with you that someone got this wrong by accident, that seems like Eliezer agrees with me that someone identifying as a Rationalist has to be trying to get core things wrong to end up saying something like that.
I don’t think this follows. I do not see how degree of wrongness implies intent. Eliezer’s comment rhetorically suggests intent (“trolling”) as a way of highlighting how wrong the person is; he is free to correct me if I am wrong, but I am pretty sure that is not an actual suggestion of intent, only a rhetorical one.
I would say moreover, that this is the sort of mistake that occurs, over and over, by default, with no intent necessary. I might even say that it is avoiding, not committing, this sort of mistake, that requires intent. Because this sort of mistake is just sort of what people fall into by default, and avoiding it requires active effort.
Is it contrary to everything Eliezer’s ever written? Sure! But reading the entirety of the Sequences, calling yourself a “rationalist”, does not in any way obviate the need to do the actual work of better group epistemology, of noticing such mistakes (and the path to them) and correcting/avoiding them.
I think we can only infer intent like you’re talking about if the person in question is, actually, y’know, thinking about what they’re doing. But I think people are really, like, acting on autopilot a pretty big fraction of the time; not autopiloting takes effort, and doing that work may be what a “rationalist” is supposed to do, it’s still not the default. All I think we can infer from this is a failure to do the work to shift out of autopilot and think. Bad group epistemology via laziness rather than via intent strikes me as the more likely explanation.
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
Once is happenstance. Twice is coincidence. Three times is enemy action.
I can imagine, after reading the sequences, continuing to have the epistemic modesty bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
Behavior is better explained as strategy than as error, if the behaviors add up to push the world in some direction (along a dimension that’s “distant” from the behavior, like how “make a box with food appear at my door” is “distant” from “wiggle my fingers on my keyboard”). If a pattern of correlated error is the sort of pattern that doesn’t easily push the world in a direction, then that pattern might be evidence against intent. For example, the conjunction fallacy will produce a pattern of wrong probability estimates with a distinct character, but it seems unlikely to push the world in some specific direction (beyond whatever happens when you have incoherent probabilities). (Maybe this argument is fuzzy on the edges, like if someone keeps trying to show you information and you keep ignoring it, you’re sort of “pushing the world in a direction” when compared to what’s “supposed to happen”, i.e. that you update; which suggests intent, although it’s “reactive” rather than “proactive”, whatever that means. I at least claim that your argument is too general, proves too much, and would be more clear if it were narrower.)
The effective direction the epistemic modesty / argument from authority bias pushes things, is away from shared narrative as something that dynamically adjusts to new information, and towards shared narrative as a way to identify and coordinate who’s receiving instructions from whom.
People frequently make “mistakes” as a form of submission, and it shouldn’t be surprising that other types of systematic error function as a means of domination, i.e. of submission enforcement.
If someone makes correlated errors, they are better explained as part of a strategy.
That does seem right to me.
It seems like very often correlated errors are the result of a mistaken, upstream crux. They’re making one mistake, which is flowing into a bunch of specific instances.
This at least has to be another hypothesis, along with “this is a conscious or unconscious strategy to get what they want.”
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
I mean, there is a word for correlated errors, and that word is “bias”; so you seem to be essentially claiming that people are unbiased? I’m guessing that’s probably not what you’re trying to claim, but that is what I am concluding? Regardless, I’m saying people are biased towards this mistake.
Or really, what I’m saying it’s the same sort of phenomenon that Eliezer discusses here. So it could indeed be construed as a strategy as you say; but it would not be a strategy on the part of the conscious agent, but rather a strategy on the part of the “corrupted hardware” itself. Or something like that—sorry, that’s not a great way of putting it, but I don’t really have a better one, and I hope that conveys what I’m getting at.
Like, I think you’re assuming too much awareness/agency of people. A person who makes correlated errors, and is aware of what they are doing, is executing a deliberate strategy. But lots of people who make correlated errors are just biased, or the errors are part of a built-in strategy they’re executing, not deliberately, but by default without thinking about it, that requires effort not to execute.
We should expect someone calling themself a rationalist to be better, obviously, but, IDK, sometimes things go bad?
I can imagine, after reading the sequences, continuing to have this bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
I mean people don’t necessarily fully internalize everything they read, and in some people the “hold on what am I doing?” can be weak? <shrug>
I mean I certainly don’t want to rule out deliberate malice like you’re talking about, but neither do I think this one snippet is enough to strongly conclude it.
In most cases it seems intentional but not deliberate. People will resist pressure to change the pattern, or find new ways to execute it if the specific way they were engaged in this bias is effectively discouraged, but don’t consciously represent to themselves their intent to do it or engage in explicit means-ends reasoning about it.
Yeah, that sounds about right to me. I’m not saying that you should assume such people are harmless or anything! Just that, like, you might want to try giving them a kick first—“hey, constant vigilance, remember?” :P—and see how they respond before giving up and treating them as hostile.
“How dare you think that you’re better at meta-rationality than Eliezer Yudkowsky, do you think you’re special” reads to me as something Eliezer Yudkowsky himself would never write.
You also wrote a whole screed about how anyone who attacks you or Scott Alexander is automatically an evil person with no ethics, and walked it back only after backlash and only halfway. You don’t get to pretend you’re exactly embracing criticism there, Yud—in fact, it was that post that severed my ties to this community for good.
FWIW I believe “Yud” is a dispreferred term (because it’s predominantly used by sneering critics), and your comment wouldn’t have gotten so many downvotes without it.
Something I try to keep in mind about critics is that people who deeply disagree with you are also not usually very invested in what you’re doing, so from their perspective there isn’t much of an incentive to put effort into their criticism. But in theory, the people who disagree with you the most are also the ones you can learn the most from.
You want to be the sort of person where if you’re raised Christian, and an atheist casually criticizes Christianity, you don’t reject the criticism immediately because “they didn’t even take the time to read the Bible!”
I think I have a lot less (true, useful, action-relevant) stuff to learn from a random fundamentalist Christian than from Carl Shulman, even though I disagree vastly more with the fundamentalist than I do with Carl.
Prescriptive appropriateness vs. descriptive appropriateness.
ESRogs is pointing out a valuable item in a civilizing peace treaty; an available weapon that, if left unused, allows a greater set of possible cooperations to come into existence. “Not appropriate” as a normative/hopeful statement, signaling his position as a signatory to that disarmament clause and one who hopes LW has already or will also sign on, as a subculture.
Zack is pointing out that, from the inside of a slur, it has precisely the purpose that ESRogs is labeling inappropriate. For a slur to express hostility and disregard is like a hammer being used to pound nails. “Entirely appropriate” as a descriptive/technical statement.
I think it would have been better if Zack had made that distinction, which I think he’s aware of, but I’m happy to pop in to help; I suspect meeting that bar would’ve prevented him from saying anything at all in this case, which would probably have been worse overall.
The rant (now somewhat redacted) can be found here, in response to the leaked emails of Scott more-or-less outright endorsing people like Steve Sailer re:”HBD”. There was a major backlash against Scott at the time, resulting in the departure of many longtime members of the community (including me), and Eliezer’s post was in response to that. It opened with:
it should have been obvious to anyone at this point that anybody who openly hates on this community generally or me personally is probably also a bad person inside and has no ethics
I think it would be good to acknowledge here Eliezer’s edits. Like, do you think that
The thing you quote here is a reasonable way to communicate the thing Eliezer was trying to communicate, and that thing is absurd?
The thing you quoted is absurd, but not what Eliezer was trying to communicate, but the thing he was actually trying to communicate was also absurd?
The thing Eliezer was trying to communicate is defensible, but it’s ridiculous that he initially tried to communicate it using those words?
What Eliezer initially said was a reasonable way to communicate what he meant, and his attempts to “clarify” are actually lying about what he meant?
Something else?
Idk, I don’t want to be like “I’m fine with criticism but it’s not really valid unless you deliver it standing on one foot in a bed of hot coals as is our local custom”. And I think it’s good that you brought this post into the conversation, it’s definitely relevant to questions like how much does Eliezer tolerate criticism. No obligation on you to reply further, certainly. (And I don’t commit to replying myself, if you do, so I guess take that into account when deciding if you will or not.)
But also… like, those edits really did happen, and I think they do change a lot.
I’m not sure how I feel about the post myself, there’s definitely things like “I actually don’t know if I can pick out the thing you’re trying to point out” and “how confident are you you’re being an impartial judge of the thing when it’s directed at you”. I definitely don’t think it’s a terrible post.
But I don’t know what about the post you’re reacting to, so like, I don’t know if we’re reacting different amounts to similar things, or you’re reacting to things I think are innocuous, or you’re reacting to things I’m not seeing (either because they’re not there or because I have a blind spot), or what.
(The very first version was actually ”...openly hates on Eliezer is probably...”, which I think is, uh, more in need of revision than the one you quoted.)
I think it would be good to acknowledge here Eliezer’s edits.
I don’t. He made them only after ingroup criticism, and that only happened because it was so incredibly egregious. Remember, this was the LAST straw for me—not the first.
The thing about ingroupy status-quo bias is that you’ll justify one small thing after another, but when you get a big one-two punch—enough to shatter that bias and make you look at things from outside—your beliefs about the group can shift very rapidly. I had already been kind of leery about a number of things I’d seen, but the one-two-three punch of the Scott emails, Eliezer’s response, and the complete refusal of anyone I knew in the community to engage with these things as a problem, was that moment for me.
Even if I did give him credit for the edit—which I don’t, really—it was only the breaking point, not the sole reason I left.
I believe Eliezer about his intended message, though I think it’s right to dock him some points for phrasing it poorly—being insufficiently self-critical is an attractor that idea-based groups have to be careful of, so if there’s a risk of some people misinterpreting you as saying ‘don’t criticize the ingroup’, you should at least take the time to define what you mean by “hating on”, or give examples of the kinds of Topher-behaviors you have in mind.
There’s a different attractor I’m worried about, which is something like “requiring community leaders to walk on eggshells all the time with how they phrase stuff, asking them to strictly optimize for impact/rhetoric/influence over blurting out what they actually believe, etc.” I think it’s worth putting effort into steering away from that outcome. But I think it’s possible to be extra-careful about ‘don’t criticize the ingroup’ signals without that spilling over into a generally high level of self-censorship.
You can avoid both by not having leaders who believe in terrible things (like “black people are genetically too stupid to govern themselves”) that they have to hide behind a veil of (im)plausible deniability.
Hm, so. Even just saying you don’t give him credit for the edits is at least a partial acknowledgement in my book, if you actually mean “no credit” and not “a little credit but not enough”. It helps narrow down where we disagree, because I do give him credit for them—I think it would be better if he’d started with the current version, but I think it would be much worse if he’d stopped with the first version.
But also, I guess I still don’t really know what makes this a straw for you, last or otherwise. Like I don’t know if it would still be a straw if Eliezer had started with the current version. And I don’t really have a sense of what you think Eliezer thinks. (Or if you think “what Eliezer thinks” is even a thing it’s sensible to try to talk about.) It seems you think this was really bad[1], worse than Rob’s criticism (which I think I agree with) would suggest. But I don’t know why you think that.
Which, again. No obligation to share, and I think what you’ve already shared is an asset to the conversation. But that’s where I’m at.
[1]: I get this impression from your earlier comments. Describing it as “the last straw” kind of makes it sound like not a big deal individually, but I don’t think that’s what you intended?
It would still be a straw if it started with the current version, because it is defending Scott for holding positions and supporting people I find indefensible. The moment someone like Steve Sailer is part of your “general theory of who to listen to”, you’re intellectually dead to me.
The last straw for me is that the community didn’t respond to that with “wow, Scott’s a real POS, time to distance ourselves from him and diagnose why we ever thought he was someone we wanted around”. Instead, it responded with “yep that sounds about right”. Which means the community is as indefensible as Scott is. And Eliezer, specifically, doing it meant that it wasn’t even a case of “well maybe the rank and file have some problems but at least the leadership...”
Thanks. After thinking for a bit… it doesn’t seem to me that Topher frobnitzes Scott, so indeed Eliezer’s reaction seems inappropriately strong. Publishing emails that someone requested (and was not promised) privacy for is not an act of sadism.
I don’t recall the specific claim, just that EY’s probability mass for the claim was in the 95-99% range. The person argued that because EY disagrees with some other thoughtful people on that question, he shouldn’t have such confidence.
I think people conflate the very reasonable “I am not going to adopt your 95-99% range because other thoughtful people disagree and I have no particular reason to trust you massively more than I trust other people” with the different “the fact that other thoughtful people mean there’s no way you could arrive at 95-99% confidence” which is false. I think thoughtful people disagreeing with you is decent evidence you are wrong but can still be outweighed.
I sought a lesson we could learn from this situation, and your comment captured such a lesson well.
This is reminiscent of the message of the Dune trilogy. Frank Herbert warns about society’s tendencies to “give over every decision-making capacity” to a charismatic leader. Herbert said in 1979:
The bottom line of the Dune trilogy is: beware of heroes. Much better rely on your own judgment, and your own mistakes.
If you’re not disagreeing with people about important things then you’re not thinking.
Indeed. And if the people object against someone disagreeing with them, that would imply they are 100% certain about being right.
I recently overheard someone (who I’d not met before) telling Eliezer Yudkowsky that he’s not allowed to have extreme beliefs about AGI outcomes.
On one hand, this suggests that the pressure to groupthink is strong. On the other hand, this is evidence of Eliezer not being treated as an infallible leader… which I suppose is a good news in this avalanche of bad news.
(There is a method to reduce group pressure, by making everyone write their opinion first, and only then tell each other the opinions. Problem is, this stops working if you estimate the same thing repeatedly, because people already know what the group opinion was in the past.)
Just zooming in on this, which stood out to me personally as a particular thing I’m really tired of.
If you’re not disagreeing with people about important things then you’re not thinking. There are many options for how to negotiate a significant disagreement with a colleague, including spending lots of time arguing about it, finding a compromise action, or stopping collaborating with the person (if it’s a severe disagreement, which often it can be). But telling someone that by disagreeing they’re claiming to be ‘better’ than another person in some way always feels to me like an attempt to ‘control’ the speech and behavior of the person you’re talking to, and I’m against it.
It happens a lot. I recently overheard someone (who I’d not met before) telling Eliezer Yudkowsky that he’s not allowed to have extreme beliefs about AGI outcomes. I don’t recall the specific claim, just that EY’s probability mass for the claim was in the 95-99% range. The person argued that because EY disagrees with some other thoughtful people on that question, he shouldn’t have such confidence.
(At the time, I noticed I didn’t have to be around or listen to that person and just wandered away. Poor Eliezer stayed and tried to give a thoughtful explanation for why the argument seemed bad.)
I noticed this too. I thought a bunch of people were affected by it in a sort of herd behavior way (not focused so much on MIRI/CFAR, I’m talking more broadly in the rationality/EA communities). I do think key parts of the arguments about how to think about timelines and takeoff are accurate (e.g. 1, 2), but I feel like many people weren’t making decisions because of reasons; instead they noticed their ‘leaders’ were acting scared and then they also acted scared, like a herd.
In both the Leverage situation and the AI timelines situation, I felt like nobody involved was really appreciating how much fuckery the information siloing was going to cause (and did cause) to the way the individuals in the ecosystem made decisions.
This was one of the main motivations behind my choice of example in the opening section of my 3.5 yr old post A Sketch of Good Communication btw (a small thing but still meant to openly disagree with the seeming consensus that timelines determined everything). And then later I wrote about the social dynamics a bunch more 2yrs ago when trying to expand on someone else’s question on the topic.
I affirm the correctness of Ben Pace’s anecdote about what he recently heard someone tell me.
“How dare you think that you’re better at meta-rationality than Eliezer Yudkowsky, do you think you’re special”—is somebody trolling? Have they never read anything I’ve written in my entire life? Do they have no sense, even, of irony? Yeah, sure, it’s harder to be better at some things than me, sure, somebody might be skeptical about that, but then you ask for evidence or say “Good luck proving that to us all eventually!” You don’t be like, “Do you think you’re special?” What kind of bystander-killing argumentative superweapon is that? What else would it prove?
I really don’t know how I could make this any clearer. I wrote a small book whose second half was about not doing exactly this. I am left with a sense that I really went to some lengths to prevent this, I did what society demands of a person plus over 10,000% (most people never write any extended arguments against bad epistemology at all, and society doesn’t hold that against them), I was not subtle. At some point I have to acknowledge that other human beings are their own people and I cannot control everything they do—and I hope that others will also acknowledge that I cannot avert all the wrong thoughts that other people think, even if I try, because I sure did try. A lot. Over many years. Aimed at that specific exact way of thinking. People have their own wills, they are not my puppets, they are still not my puppets even if they have read some blog posts of mine or heard summaries from somebody else who once did; I have put in at least one hundred times the amount of effort that would be required, if any effort were required at all, to wash my hands of this way of thinking.
The irony was certainly not lost on me; I’ve edited the post to make this clearer to other readers.
I’m glad you agree that the behavior Jessica describes is explicitly opposed to the content of the Sequences, and that you clearly care a lot about this. I don’t think anyone can reasonably claim you didn’t try hard to get people to behave better, or could reasonably blame you for the fact that many people persistently try to do the opposite of what you say, in the name of Rationality.
I do think it would be a very good idea for you to investigate why & how the institutions you helped build and are still actively participating in are optimizing against your explicitly stated intentions. Anna’s endorsement of this post seems like reasonably strong confirmation that organizations nominally committed to your agenda are actually opposed to it, unless you’re actually checking. And MIRI/CFAR donors seem to for the most part think that you’re aware of and endorse those orgs’ activities.
When Jessica and another recent MIRI employee asked a few years ago for some of your time to explain why they’d left, your response was:
You responded a little bit by email, but didn’t seem very interested in what was going on (you didn’t ask the kinds of followup questions that would establish common knowledge about agreement or disagreement), so your interlocutors didn’t perceive a real opportunity to inform you of these dynamics at that time.
Presumably Eliezer’s agenda is much broader than “make sure nobody tries to socially enforce deferral to high-status figures in an ungrounded way” though I do think this is part of his goals.
The above seems to me like it tries to equivocate between “this is confirmation that at least some people don’t act in full agreement with your agenda, despite being nominally committed to it” and “this is confirmation that people are actively working against your agenda”. These two really don’t strike me as the same, and I really don’t like how this comment seems like it tries to equivocate between the two.
Of course, the claim that some chunk of the community/organizations Eliezer created are working actively against some agenda that Eliezer tried to set for them is plausible. But calling the above a “strong confirmation” of this fact strikes me as a very substantial stretch.
It’s explicitly opposition to core Sequences content, which Eliezer felt was important enough to write a whole additional philosophical dialogue about after the main Sequences were done. Eliezer’s response when informed about it was:
That doesn’t seem like Eliezer agrees with you that someone got this wrong by accident, that seems like Eliezer agrees with me that someone identifying as a Rationalist has to be trying to get core things wrong to end up saying something like that.
I don’t think this follows. I do not see how degree of wrongness implies intent. Eliezer’s comment rhetorically suggests intent (“trolling”) as a way of highlighting how wrong the person is; he is free to correct me if I am wrong, but I am pretty sure that is not an actual suggestion of intent, only a rhetorical one.
I would say moreover, that this is the sort of mistake that occurs, over and over, by default, with no intent necessary. I might even say that it is avoiding, not committing, this sort of mistake, that requires intent. Because this sort of mistake is just sort of what people fall into by default, and avoiding it requires active effort.
Is it contrary to everything Eliezer’s ever written? Sure! But reading the entirety of the Sequences, calling yourself a “rationalist”, does not in any way obviate the need to do the actual work of better group epistemology, of noticing such mistakes (and the path to them) and correcting/avoiding them.
I think we can only infer intent like you’re talking about if the person in question is, actually, y’know, thinking about what they’re doing. But I think people are really, like, acting on autopilot a pretty big fraction of the time; not autopiloting takes effort, and doing that work may be what a “rationalist” is supposed to do, it’s still not the default. All I think we can infer from this is a failure to do the work to shift out of autopilot and think. Bad group epistemology via laziness rather than via intent strikes me as the more likely explanation.
This seems exactly backwards, if someone makes uncorrelated errors, they are probably unintentional mistakes. If someone makes correlated errors, they are better explained as part of a strategy.
I can imagine, after reading the sequences, continuing to have the epistemic modesty bias in my own thoughts, but I don’t see how I could have been so confused as to refer to it in conversation as a valid principle of epistemology.
Behavior is better explained as strategy than as error, if the behaviors add up to push the world in some direction (along a dimension that’s “distant” from the behavior, like how “make a box with food appear at my door” is “distant” from “wiggle my fingers on my keyboard”). If a pattern of correlated error is the sort of pattern that doesn’t easily push the world in a direction, then that pattern might be evidence against intent. For example, the conjunction fallacy will produce a pattern of wrong probability estimates with a distinct character, but it seems unlikely to push the world in some specific direction (beyond whatever happens when you have incoherent probabilities). (Maybe this argument is fuzzy on the edges, like if someone keeps trying to show you information and you keep ignoring it, you’re sort of “pushing the world in a direction” when compared to what’s “supposed to happen”, i.e. that you update; which suggests intent, although it’s “reactive” rather than “proactive”, whatever that means. I at least claim that your argument is too general, proves too much, and would be more clear if it were narrower.)
The effective direction the epistemic modesty / argument from authority bias pushes things, is away from shared narrative as something that dynamically adjusts to new information, and towards shared narrative as a way to identify and coordinate who’s receiving instructions from whom.
People frequently make “mistakes” as a form of submission, and it shouldn’t be surprising that other types of systematic error function as a means of domination, i.e. of submission enforcement.
(I indeed find this a more clear+compelling argument and appreciate you trying to make this known.)
That does seem right to me.
It seems like very often correlated errors are the result of a mistaken, upstream crux. They’re making one mistake, which is flowing into a bunch of specific instances.
This at least has to be another hypothesis, along with “this is a conscious or unconscious strategy to get what they want.”
I mean, there is a word for correlated errors, and that word is “bias”; so you seem to be essentially claiming that people are unbiased? I’m guessing that’s probably not what you’re trying to claim, but that is what I am concluding? Regardless, I’m saying people are biased towards this mistake.
Or really, what I’m saying it’s the same sort of phenomenon that Eliezer discusses here. So it could indeed be construed as a strategy as you say; but it would not be a strategy on the part of the conscious agent, but rather a strategy on the part of the “corrupted hardware” itself. Or something like that—sorry, that’s not a great way of putting it, but I don’t really have a better one, and I hope that conveys what I’m getting at.
Like, I think you’re assuming too much awareness/agency of people. A person who makes correlated errors, and is aware of what they are doing, is executing a deliberate strategy. But lots of people who make correlated errors are just biased, or the errors are part of a built-in strategy they’re executing, not deliberately, but by default without thinking about it, that requires effort not to execute.
We should expect someone calling themself a rationalist to be better, obviously, but, IDK, sometimes things go bad?
I mean people don’t necessarily fully internalize everything they read, and in some people the “hold on what am I doing?” can be weak? <shrug>
I mean I certainly don’t want to rule out deliberate malice like you’re talking about, but neither do I think this one snippet is enough to strongly conclude it.
In most cases it seems intentional but not deliberate. People will resist pressure to change the pattern, or find new ways to execute it if the specific way they were engaged in this bias is effectively discouraged, but don’t consciously represent to themselves their intent to do it or engage in explicit means-ends reasoning about it.
Yeah, that sounds about right to me. I’m not saying that you should assume such people are harmless or anything! Just that, like, you might want to try giving them a kick first—“hey, constant vigilance, remember?” :P—and see how they respond before giving up and treating them as hostile.
“How dare you think that you’re better at meta-rationality than Eliezer Yudkowsky, do you think you’re special” reads to me as something Eliezer Yudkowsky himself would never write.
You also wrote a whole screed about how anyone who attacks you or Scott Alexander is automatically an evil person with no ethics, and walked it back only after backlash and only halfway. You don’t get to pretend you’re exactly embracing criticism there, Yud—in fact, it was that post that severed my ties to this community for good.
FWIW I believe “Yud” is a dispreferred term (because it’s predominantly used by sneering critics), and your comment wouldn’t have gotten so many downvotes without it.
I strong-downvoted because because they didn’t bother to even link to the so-called screed. (Forgive me for not blindly trusting throwaway46237896.)
Something I try to keep in mind about critics is that people who deeply disagree with you are also not usually very invested in what you’re doing, so from their perspective there isn’t much of an incentive to put effort into their criticism. But in theory, the people who disagree with you the most are also the ones you can learn the most from.
You want to be the sort of person where if you’re raised Christian, and an atheist casually criticizes Christianity, you don’t reject the criticism immediately because “they didn’t even take the time to read the Bible!”
I think I have a lot less (true, useful, action-relevant) stuff to learn from a random fundamentalist Christian than from Carl Shulman, even though I disagree vastly more with the fundamentalist than I do with Carl.
The original “sneer club” comment?
Really? I do it because it’s easier to type. Maybe I’m missing some historical context here.
For some reason a bunch of people started referring to him as “Big Yud” on Twitter. Here’s some context regarding EY’s feelings about it.
I’m a former member turned very hostile to the community represented here these days. So that’s appropriate, I guess.
Any thoughts on how we can help you be at peace?
I disagree that it’s appropriate to use terms for people that they consider slurs because they’re part of a community that you don’t like.
It’s entirely appropriate! Expressing hostility is what slurs are for!
Prescriptive appropriateness vs. descriptive appropriateness.
ESRogs is pointing out a valuable item in a civilizing peace treaty; an available weapon that, if left unused, allows a greater set of possible cooperations to come into existence. “Not appropriate” as a normative/hopeful statement, signaling his position as a signatory to that disarmament clause and one who hopes LW has already or will also sign on, as a subculture.
Zack is pointing out that, from the inside of a slur, it has precisely the purpose that ESRogs is labeling inappropriate. For a slur to express hostility and disregard is like a hammer being used to pound nails. “Entirely appropriate” as a descriptive/technical statement.
I think it would have been better if Zack had made that distinction, which I think he’s aware of, but I’m happy to pop in to help; I suspect meeting that bar would’ve prevented him from saying anything at all in this case, which would probably have been worse overall.
What screed are you referring to?
The rant (now somewhat redacted) can be found here, in response to the leaked emails of Scott more-or-less outright endorsing people like Steve Sailer re:”HBD”. There was a major backlash against Scott at the time, resulting in the departure of many longtime members of the community (including me), and Eliezer’s post was in response to that. It opened with:
...which is, to put it mildly, absurd.
I think it would be good to acknowledge here Eliezer’s edits. Like, do you think that
The thing you quote here is a reasonable way to communicate the thing Eliezer was trying to communicate, and that thing is absurd?
The thing you quoted is absurd, but not what Eliezer was trying to communicate, but the thing he was actually trying to communicate was also absurd?
The thing Eliezer was trying to communicate is defensible, but it’s ridiculous that he initially tried to communicate it using those words?
What Eliezer initially said was a reasonable way to communicate what he meant, and his attempts to “clarify” are actually lying about what he meant?
Something else?
Idk, I don’t want to be like “I’m fine with criticism but it’s not really valid unless you deliver it standing on one foot in a bed of hot coals as is our local custom”. And I think it’s good that you brought this post into the conversation, it’s definitely relevant to questions like how much does Eliezer tolerate criticism. No obligation on you to reply further, certainly. (And I don’t commit to replying myself, if you do, so I guess take that into account when deciding if you will or not.)
But also… like, those edits really did happen, and I think they do change a lot.
I’m not sure how I feel about the post myself, there’s definitely things like “I actually don’t know if I can pick out the thing you’re trying to point out” and “how confident are you you’re being an impartial judge of the thing when it’s directed at you”. I definitely don’t think it’s a terrible post.
But I don’t know what about the post you’re reacting to, so like, I don’t know if we’re reacting different amounts to similar things, or you’re reacting to things I think are innocuous, or you’re reacting to things I’m not seeing (either because they’re not there or because I have a blind spot), or what.
(The very first version was actually ”...openly hates on Eliezer is probably...”, which I think is, uh, more in need of revision than the one you quoted.)
Strong approval for the way this comment goes about making its point, and trying to bridge the inferential gap.
I don’t. He made them only after ingroup criticism, and that only happened because it was so incredibly egregious. Remember, this was the LAST straw for me—not the first.
The thing about ingroupy status-quo bias is that you’ll justify one small thing after another, but when you get a big one-two punch—enough to shatter that bias and make you look at things from outside—your beliefs about the group can shift very rapidly. I had already been kind of leery about a number of things I’d seen, but the one-two-three punch of the Scott emails, Eliezer’s response, and the complete refusal of anyone I knew in the community to engage with these things as a problem, was that moment for me.
Even if I did give him credit for the edit—which I don’t, really—it was only the breaking point, not the sole reason I left.
I believe Eliezer about his intended message, though I think it’s right to dock him some points for phrasing it poorly—being insufficiently self-critical is an attractor that idea-based groups have to be careful of, so if there’s a risk of some people misinterpreting you as saying ‘don’t criticize the ingroup’, you should at least take the time to define what you mean by “hating on”, or give examples of the kinds of Topher-behaviors you have in mind.
There’s a different attractor I’m worried about, which is something like “requiring community leaders to walk on eggshells all the time with how they phrase stuff, asking them to strictly optimize for impact/rhetoric/influence over blurting out what they actually believe, etc.” I think it’s worth putting effort into steering away from that outcome. But I think it’s possible to be extra-careful about ‘don’t criticize the ingroup’ signals without that spilling over into a generally high level of self-censorship.
You can avoid both by not having leaders who believe in terrible things (like “black people are genetically too stupid to govern themselves”) that they have to hide behind a veil of (im)plausible deniability.
Hm, so. Even just saying you don’t give him credit for the edits is at least a partial acknowledgement in my book, if you actually mean “no credit” and not “a little credit but not enough”. It helps narrow down where we disagree, because I do give him credit for them—I think it would be better if he’d started with the current version, but I think it would be much worse if he’d stopped with the first version.
But also, I guess I still don’t really know what makes this a straw for you, last or otherwise. Like I don’t know if it would still be a straw if Eliezer had started with the current version. And I don’t really have a sense of what you think Eliezer thinks. (Or if you think “what Eliezer thinks” is even a thing it’s sensible to try to talk about.) It seems you think this was really bad[1], worse than Rob’s criticism (which I think I agree with) would suggest. But I don’t know why you think that.
Which, again. No obligation to share, and I think what you’ve already shared is an asset to the conversation. But that’s where I’m at.
[1]: I get this impression from your earlier comments. Describing it as “the last straw” kind of makes it sound like not a big deal individually, but I don’t think that’s what you intended?
It would still be a straw if it started with the current version, because it is defending Scott for holding positions and supporting people I find indefensible. The moment someone like Steve Sailer is part of your “general theory of who to listen to”, you’re intellectually dead to me.
The last straw for me is that the community didn’t respond to that with “wow, Scott’s a real POS, time to distance ourselves from him and diagnose why we ever thought he was someone we wanted around”. Instead, it responded with “yep that sounds about right”. Which means the community is as indefensible as Scott is. And Eliezer, specifically, doing it meant that it wasn’t even a case of “well maybe the rank and file have some problems but at least the leadership...”
Thanks. After thinking for a bit… it doesn’t seem to me that Topher frobnitzes Scott, so indeed Eliezer’s reaction seems inappropriately strong. Publishing emails that someone requested (and was not promised) privacy for is not an act of sadism.
I believe the idea was not that this was an act of frobnitzing, but that
Topher is someone who openly frobnitzes.
Now he’s done this, which is bad.
It is unsurprising that someone who openly frobnitzes does other bad things too.
I think people conflate the very reasonable “I am not going to adopt your 95-99% range because other thoughtful people disagree and I have no particular reason to trust you massively more than I trust other people” with the different “the fact that other thoughtful people mean there’s no way you could arrive at 95-99% confidence” which is false. I think thoughtful people disagreeing with you is decent evidence you are wrong but can still be outweighed.
I sought a lesson we could learn from this situation, and your comment captured such a lesson well.
This is reminiscent of the message of the Dune trilogy. Frank Herbert warns about society’s tendencies to “give over every decision-making capacity” to a charismatic leader. Herbert said in 1979:
This is a great sentence. I kind of want it on a t-shirt.
Indeed. And if the people object against someone disagreeing with them, that would imply they are 100% certain about being right.
On one hand, this suggests that the pressure to groupthink is strong. On the other hand, this is evidence of Eliezer not being treated as an infallible leader… which I suppose is a good news in this avalanche of bad news.
(There is a method to reduce group pressure, by making everyone write their opinion first, and only then tell each other the opinions. Problem is, this stops working if you estimate the same thing repeatedly, because people already know what the group opinion was in the past.)