I think Eliezer is just really rude and uninterested in behaving civilly, and has terrible intuitions about a wide variety of topics, especially topics related to how other people think or behave. And he substantially evaluates whether people are smart or reasonable based on how much they agree with him or respect him, and therefore writes off a lot of people and behaves contemptuously toward them. And he ends up surrounded by people who either hero worship him or understate their disagreements with him in order to get along with him—many of his co-workers would prefer he didn’t act like an asshole on the internet, but they can’t make that happen.
I think the core problem with Eliezer is that he spent his formative years arguing on the internet with people on listservs, most of whom were extremely unreasonable. And so he’s used to the people around him being mostly idiots with incredibly stupid takes and very little value to add. So he is quite unused to changing his mind based on things other people say.
I don’t think you should consider him to be rational with respect to this kind of decision. (I also don’t think you should consider him to be rational when thinking about AI.)
I personally would not recommend financial support of MIRI, because I’m worried it will amplify net negative communications from him, and I’m worried that it will cause him to have more of an effect on discourse e.g. on LessWrong. I like and respect many MIRI staff, and I think they should work elsewhere and on projects other than amplifying Eliezer.
(Eliezer is pleasant and entertaining in person if you aren’t talking about topics where he thinks your opinion is dumb. I’ve overall enjoyed interacting with him in person, and he’s generally treated me kindly in person, and obviously I’m very grateful for the work he did putting the rationalist community together.)
I personally would not recommend financial support of MIRI, because I’m worried it will amplify net negative communications from him
Small note: Eliezer is largely backing off from direct comms and most of our comms in the next year will be less Eliezer’s-direct-words-being-promoted than in the past (as opposed to more). Obviously still lots of Eliezer thoughts and Eliezer perspectives and goals, but more filtered as opposed to less so. Just FYI.
Oh alas, I think that is a major update downwards on MIRI’s work here. Happy to chat about it if you want sometime, but it appears to me that almost every time Eliezer intentionally writes substantial public comms here, things get non-trivially better (e.g. I think the Time article was much better than other things MIRI had done for a while). I am not super confident here.
To be clear, it’s not because we agree with Buck’s model. It’s more that Eliezer has persistent health and stamina issues and others (Nate, Malo, etc.) need to step up and receive the torch.
Tracking your attitudes here is pretty important to me, because I respect you a lot and also work for MIRI. Still, it’s been kind of hard, because sometimes it looks like you’re pleasantly surprised (e.g., about the first two sections of IABIED: “After reading the book, it feels like a shocking oversight that no one wrote it earlier” and “it’s hard for me to imagine someone else writing a much better [book for general audiences on x-risk]”), and then other times it looks like that pleasant surprise hasn’t propagated through your attitudes more broadly.
“The main thing Eliezer and MIRI have been doing since shifting focus to comms addressed a ‘shockingoversight’ that it’s hard to imagine anyone else doing a better job addressing” (lmk if this doesn’t feel like an accurate paraphrase) feels like it reflects a pretty strong positive update in the speaker! (especially having chatted about your views before that)
I guess I was just surprised / confused by the paragraph that starts “I personally would not...”, given the trajectory over the past few months of your impressions of MIRI’s recent work. Would you have said something much more strongly negative in August? Does IABIED not significantly inform your expectations of future MIRI outputs? Something else?
I can see why the different things I’ve said on this might seem inconsistent :P It’s also very possible I’m wrong here, I’m not confident about this and have only spent a few hours in conversation about it. And if I wasn’t recently personally angered by Eliezer’s behavior, I wouldn’t have mentioned this opinion publicly. But here’s my current model.
My current sense is that IABIED hasn’t had that much of an effect on public perception of AI risk, compared to things like AI 2027. My previous sense was that there are huge downsides of Eliezer (and co) being more influential on the topic of AI safety, but MIRI had some chance of succeeding at getting lots of attention, so I was overall positive on you and other MIRI people putting your time into promoting the book. Because the book didn’t go as well as seemed plausible, promoting Eliezer’s perspective seems less like an efficient way of popularizing concern about AI risk, and less outweighs the disadvantages of him being having negative effects inside the AI safety community.
For example, my guess is that it’s worse for the MIRI governance team to be at MIRI than elsewhere except in as much as they gain prominence due to Eliezer association; if that second factor is weaker, it looks less good for them to be there.
I think my impression of the book is somewhat more negative than it was when it first came out, based on various discussions I’ve had with people about it. But this isn’t a big factor.
Does this make sense?
“The main thing Eliezer and MIRI have been doing since shifting focus to comms addressed a ‘shockingoversight’ that it’s hard to imagine anyone else doing a better job addressing” (lmk if this doesn’t feel like an accurate paraphrase)
This paraphrase doesn’t quite preserve the meaning I intended. I think many people would have done a somewhat better job.
For example, my guess is that it’s worse for the MIRI governance team to be at MIRI than elsewhere except in as much as they gain prominence due to Eliezer association
Or if they want to work from a frame that isn’t really supported by other orgs (i.e., they’re closer to Eliezer’s views than to the views/filters enforced at AIFP, RAND, Redwood, and other alternatives). I think people at MIRI think halt/off-switch is a good idea, and want to work on it. Many (but not all) of us think it’s Our Best Hope, and would be pretty dissatisfied working on something else.
I agree that visible-impact-so-far for AI2027 is > it is for IABIED, but I’m more optimistic than you about IABIED’s impact into the future (both because I like IABIED more than you do, and because I’m keeping an eye on ongoing sales, readership, assignment in universities, etc).
I think my impression of the book is somewhat more negative than it was when it first came out, based on various discussions I’ve had with people about it.
Consider leaving a comment on your review about this if you have the time and inclination in the future; I’m at least curious, and others may be, too.
I think you probably didn’t read the moderation guidelines for this post:
Moderation Note: Please don’t comment with “sides”, eg. “Eliezer is [good]/[bad]”, “people who find him abrasive are [right]/[wrong]”.
This comment seems to me to straightforwardly violate them. To be clear, I am not saying the things you are saying here should not be said, it just seems like the author was trying to have a pretty different conversation (and my guess is the author is right that whatever macro conversation is going on here will go better if people follow these guidelines for now).
Makes sense. I think I’ll move it out of the answers into the comments but leave it around, but might delete it if it ends up dominating the rest of the conversation.
I agree with this decision. You reference the comment in one of your answers. If it starts taking over, it should be removed, but can otherwise provide interesting meta-commentary.
it just seems like the author was trying to have a pretty different conversation
I think mostly in tone. If I imagine a somewhat less triggered intro sentence in Buck’s comment, it seems to be straightforwardly motivating answers to the two questions at the end of OP:
1. None of Eliezer’s public communication is -EV for AI Safety 2. Financial support of MIRI is likely to produce more consistently +EV communication than historically seen from Eliezer individually.
ETA: I do think the OP was trying to avoid spawning demon threads, which is a good impulse to have (especially when it comes to questions like this).
Eliezer is pleasant and entertaining in person if you aren’t talking about topics where he thinks your opinion is dumb.
Except that I don’t think that I understand the distinction between Eliezer believing that the opinion is dumb versus the opinion being actually dumb. The examples which I cite in my other comment are, in my opinion, close to the latter.
However, while Eliezer did concede in cases like this or that[1], I can’t exclude the possibility that Eliezer ranted over something[2] which is verified to be not actually stupid. Alas, the two most prominent and highest-stakes examples, which are the difficulty of ASI alignment[3] and ASI takeoff speed[4], aren’t yet resolved since mankind hasn’t created any ASIs and measured the takeoff speeds.
Edited to add: The example of Yudkowsky conceding is the following. “If you doubt my ability to ever concede to evidence about this sort of topic, observe this past case on Twitter where I immediately and without argument concede that OpenPhil was right and I was wrong, the moment that the evidence appeared to be decisive. (The choice of example may seem snarky but is not actually snark; it is not easy for me to find other cases where, according to my own view, clear concrete evidence came out that I was definitely wrong and OpenPhil definitely right; and I did in that case immediately concede.)”
The most prominent candidate of which I am aware is his comments to his most recent post and his likely failure to understand “how you could build something not-BS” on Cotra’s estimates. I explained how Kokotajlo obtained a fairly good prediction, Eliezer criticized Cotra’s entire methodology instead of the parameter choice.
For reference, I have compared SOTA research with kids’ psychology and actually interesting research with aligning adults or even with solving problems like the AIs being the new proletariat.
I think Eliezer is just really rude and uninterested in behaving civilly, and has terrible intuitions about a wide variety of topics, especially topics related to how other people think or behave. And he substantially evaluates whether people are smart or reasonable based on how much they agree with him or respect him, and therefore writes off a lot of people and behaves contemptuously toward them. And he ends up surrounded by people who either hero worship him or understate their disagreements with him in order to get along with him—many of his co-workers would prefer he didn’t act like an asshole on the internet, but they can’t make that happen.
I think the core problem with Eliezer is that he spent his formative years arguing on the internet with people on listservs, most of whom were extremely unreasonable. And so he’s used to the people around him being mostly idiots with incredibly stupid takes and very little value to add. So he is quite unused to changing his mind based on things other people say.
I don’t think you should consider him to be rational with respect to this kind of decision. (I also don’t think you should consider him to be rational when thinking about AI.)
I personally would not recommend financial support of MIRI, because I’m worried it will amplify net negative communications from him, and I’m worried that it will cause him to have more of an effect on discourse e.g. on LessWrong. I like and respect many MIRI staff, and I think they should work elsewhere and on projects other than amplifying Eliezer.
(Eliezer is pleasant and entertaining in person if you aren’t talking about topics where he thinks your opinion is dumb. I’ve overall enjoyed interacting with him in person, and he’s generally treated me kindly in person, and obviously I’m very grateful for the work he did putting the rationalist community together.)
Small note: Eliezer is largely backing off from direct comms and most of our comms in the next year will be less Eliezer’s-direct-words-being-promoted than in the past (as opposed to more). Obviously still lots of Eliezer thoughts and Eliezer perspectives and goals, but more filtered as opposed to less so. Just FYI.
Oh alas, I think that is a major update downwards on MIRI’s work here. Happy to chat about it if you want sometime, but it appears to me that almost every time Eliezer intentionally writes substantial public comms here, things get non-trivially better (e.g. I think the Time article was much better than other things MIRI had done for a while). I am not super confident here.
To be clear, it’s not because we agree with Buck’s model. It’s more that Eliezer has persistent health and stamina issues and others (Nate, Malo, etc.) need to step up and receive the torch.
(Also “less” doesn’t mean “zero”.)
Tracking your attitudes here is pretty important to me, because I respect you a lot and also work for MIRI. Still, it’s been kind of hard, because sometimes it looks like you’re pleasantly surprised (e.g., about the first two sections of IABIED: “After reading the book, it feels like a shocking oversight that no one wrote it earlier” and “it’s hard for me to imagine someone else writing a much better [book for general audiences on x-risk]”), and then other times it looks like that pleasant surprise hasn’t propagated through your attitudes more broadly.
“The main thing Eliezer and MIRI have been doing since shifting focus to comms addressed a ‘shocking oversight’ that it’s hard to imagine anyone else doing a better job addressing” (lmk if this doesn’t feel like an accurate paraphrase) feels like it reflects a pretty strong positive update in the speaker! (especially having chatted about your views before that)
I guess I was just surprised / confused by the paragraph that starts “I personally would not...”, given the trajectory over the past few months of your impressions of MIRI’s recent work. Would you have said something much more strongly negative in August? Does IABIED not significantly inform your expectations of future MIRI outputs? Something else?
I can see why the different things I’ve said on this might seem inconsistent :P It’s also very possible I’m wrong here, I’m not confident about this and have only spent a few hours in conversation about it. And if I wasn’t recently personally angered by Eliezer’s behavior, I wouldn’t have mentioned this opinion publicly. But here’s my current model.
My current sense is that IABIED hasn’t had that much of an effect on public perception of AI risk, compared to things like AI 2027. My previous sense was that there are huge downsides of Eliezer (and co) being more influential on the topic of AI safety, but MIRI had some chance of succeeding at getting lots of attention, so I was overall positive on you and other MIRI people putting your time into promoting the book. Because the book didn’t go as well as seemed plausible, promoting Eliezer’s perspective seems less like an efficient way of popularizing concern about AI risk, and less outweighs the disadvantages of him being having negative effects inside the AI safety community.
For example, my guess is that it’s worse for the MIRI governance team to be at MIRI than elsewhere except in as much as they gain prominence due to Eliezer association; if that second factor is weaker, it looks less good for them to be there.
I think my impression of the book is somewhat more negative than it was when it first came out, based on various discussions I’ve had with people about it. But this isn’t a big factor.
Does this make sense?
This paraphrase doesn’t quite preserve the meaning I intended. I think many people would have done a somewhat better job.
Or if they want to work from a frame that isn’t really supported by other orgs (i.e., they’re closer to Eliezer’s views than to the views/filters enforced at AIFP, RAND, Redwood, and other alternatives). I think people at MIRI think halt/off-switch is a good idea, and want to work on it. Many (but not all) of us think it’s Our Best Hope, and would be pretty dissatisfied working on something else.
I agree that visible-impact-so-far for AI2027 is > it is for IABIED, but I’m more optimistic than you about IABIED’s impact into the future (both because I like IABIED more than you do, and because I’m keeping an eye on ongoing sales, readership, assignment in universities, etc).
Consider leaving a comment on your review about this if you have the time and inclination in the future; I’m at least curious, and others may be, too.
(probably I bow out now; thanks Buck!)
I think you probably didn’t read the moderation guidelines for this post:
This comment seems to me to straightforwardly violate them. To be clear, I am not saying the things you are saying here should not be said, it just seems like the author was trying to have a pretty different conversation (and my guess is the author is right that whatever macro conversation is going on here will go better if people follow these guidelines for now).
FWIW I almost missed the moderation guidelines for this post, it’s rare that people actually edit them.
Fair enough! Agree it’s not super widely used, but still seems like we should enforce it when people do use them.
Oh, you’re right, I didn’t read those. Feel free to remove the comment or whatever you think is the right move.
Makes sense. I think I’ll move it out of the answers into the comments but leave it around, but might delete it if it ends up dominating the rest of the conversation.
I agree with this decision. You reference the comment in one of your answers. If it starts taking over, it should be removed, but can otherwise provide interesting meta-commentary.
I think mostly in tone. If I imagine a somewhat less triggered intro sentence in Buck’s comment, it seems to be straightforwardly motivating answers to the two questions at the end of OP:
ETA: I do think the OP was trying to avoid spawning demon threads, which is a good impulse to have (especially when it comes to questions like this).
How does the intro sentence seem triggered? How would you have written it?
Except that I don’t think that I understand the distinction between Eliezer believing that the opinion is dumb versus the opinion being actually dumb. The examples which I cite in my other comment are, in my opinion, close to the latter.
However, while Eliezer did concede in cases like this or that[1], I can’t exclude the possibility that Eliezer ranted over something[2] which is verified to be not actually stupid. Alas, the two most prominent and highest-stakes examples, which are the difficulty of ASI alignment[3] and ASI takeoff speed[4], aren’t yet resolved since mankind hasn’t created any ASIs and measured the takeoff speeds.
Edited to add: The example of Yudkowsky conceding is the following. “If you doubt my ability to ever concede to evidence about this sort of topic, observe this past case on Twitter where I immediately and without argument concede that OpenPhil was right and I was wrong, the moment that the evidence appeared to be decisive. (The choice of example may seem snarky but is not actually snark; it is not easy for me to find other cases where, according to my own view, clear concrete evidence came out that I was definitely wrong and OpenPhil definitely right; and I did in that case immediately concede.)”
The most prominent candidate of which I am aware is his comments to his most recent post and his likely failure to understand “how you could build something not-BS” on Cotra’s estimates. I explained how Kokotajlo obtained a fairly good prediction, Eliezer criticized Cotra’s entire methodology instead of the parameter choice.
For reference, I have compared SOTA research with kids’ psychology and actually interesting research with aligning adults or even with solving problems like the AIs being the new proletariat.
Which Yudkowsky assumes to be very fast and the AI-2027 forecast assumes to be rather slow: the scenario had Agent-4 become adversarial in September 2027, solve mechinterp and create Agent-5 in November 2027; by June 2028 Agent-5 was thought to become wildly superintelligent.