based on the claim that while nearly everyone else was criminally insane (causing huge amounts of damage due to disconnect from reality, in a way that would be criminal if done knowingly), he, almost uniquely, was not.
I don’t think Eliezer has ever made this claim in the sense that you connote [...]
I read Yudkowsky as asserting:
A very high estimate of his own intelligence, eg comparable to Feynman and Hofstadter.
A very high estimate of the value of intelligence in general, eg sufficient to takeover the world and tile the lightcone using only an internet connection.
A very high estimate of the damage caused on Earth by insufficient intelligence, eg human extinction.
In the fictional world of Dath Ilan where Yudkowsky is the median inhabitant, Yudkowsky says they are on track to solve the alignment (~95% confidence). Whereas in the actual world he says we are on track to go extinct (~100% confidence). Causing human extinction would be criminal if done knowingly, so this satisfies Zach’s claim as written.
I’m leaving this light on links, because I’m not sure what of the above you might object to. I realize that you had many other objections to Zach’s framing, but I thought this could be something to drill into.
Edit: I’m not offering any money to respond, and independently of that, it’s 100% fine if you don’t want to respond.
I mostly take issue with the phrases “criminally insane”, “causing enormous damage”, and “he, almost uniquely, was not” connoting a more unusual and more actionable view than Eliezer (or almost anyone else) actually holds or would agree with.
Lots of people in the world are doing things that are straightforwardly non-optimal, often not even in their own narrow self-interest. This is mostly just the mistake side of conflict vs. mistake theory though, which seems relatively uncontroversial, at least on LW.
Eliezer has pointed out some of those mistakes in the context of AGI and other areas, but so have many others (Scott Alexander, Zack himself, etc.), in a variety of areas (housing policy, education policy, economics, etc.). Such explanations often come (implicitly or explicitly) with a call for others to change their behavior if they accept such arguments, but Eliezer doesn’t seem particularly strident in making such calls, compared to e.g. ordinary politicians, public policy advocates, or other rationalists.
Note, I’m not claiming that Eliezer does not hold some object-level views considered weird or extreme by most, e.g. that sufficiently intelligent AGI could take over the world, or that establishing multinational agreements for control and use of GPUs would be good policy.
But I agree with most or all those views because they seem correct on the merits, not because Eliezer (or anyone else) happened to say them. Eliezer may have been the one to point them out, for which he deserves credit, but he’s always explained his reasoning (sometimes at extreme length) and never to my knowledge asked anyone to just trust him about something like that and start making drastic behavioral changes as a result.
In the fictional world of Dath Ilan where Yudkowsky is the median inhabitant, Yudkowsky says they are on track to solve the alignment (~95% confidence). Whereas in the actual world he says we are on track to go extinct (~100% confidence). Causing human extinction would be criminal if done knowingly, so this satisfies Zach’s claim as written.
Again, leaving aside whether Eliezer himself has actually claimed this or would agree with the sentiment, it seems straightforwardly true to me that a world where the median IQ was 140 would look a lot different (and be a lot better off) than the current Earth. Whether or not it would look exactly like Eliezer’s speculative fictional world, it seems strange and uncharitable to me to characterize that view (that the world would be much better off with more intelligence) as extreme, or to interpret it as a demand or call to action for anything in particular.
Zach (et al) are exaggerating how unusual/extreme/weird Yudkowsky’s positions are.
Zach (et al) are exaggerating how much Yudkowsky’s writings are an explicit call to action.
To the extent that Yudkowsky has unusual positions and calls for actions, you think he’s mostly correct on the merits.
Of these, I’d like to push on (1) a bit. However, I think this would probably work better as a new top-level post (working title “Yudkowsky on Yudkowsky”). To give a flavor, though, and because I’m quite likely to fail to write the top-level post, here’s an example. Shah and Yudkowsky on alignment failures.
This may, perhaps, be confounded by the phenomenon where I am one of the last living descendants of the lineage that ever knew how to say anything concrete at all. Richard Feynman—or so I would now say in retrospect—is noticing concreteness dying out of the world, and being worried about that, at the point where he goes to a college and hears a professor talking about “essential objects” in class, and Feynman asks “Is a brick an essential object?”—meaning to work up to the notion of the inside of a brick, which can’t be observed because breaking a brick in half just gives you two new exterior surfaces—and everybody in the classroom has a different notion of what it would mean for a brick to be an essential object.
I encourage you to follow the link to the rest of the conversation, which relates this to alignment work. So we have this phenomenon where one factor in humanity going extinct is that people don’t listen enough to Yudkowsky and his almost unique ability to speak concretely. This also supports (2) above—this isn’t an explicit call to action, he’s just observing a phenomenon.
A (1) take here is that the quote is cherry-picked, a joke, or an outlier, and his overall work implies a more modest self-assessment. A (3) take is that he really is almost uniquely able to speak concretely. My take (4) is that his self-assessment is positively biased. I interpret Zack’s “break-up” with Yudkowsky in the opening post as moving from a (3) model to a (4) model, and encouraging others to do the same.
I read Yudkowsky as asserting:
A very high estimate of his own intelligence, eg comparable to Feynman and Hofstadter.
A very high estimate of the value of intelligence in general, eg sufficient to takeover the world and tile the lightcone using only an internet connection.
A very high estimate of the damage caused on Earth by insufficient intelligence, eg human extinction.
In the fictional world of Dath Ilan where Yudkowsky is the median inhabitant, Yudkowsky says they are on track to solve the alignment (~95% confidence). Whereas in the actual world he says we are on track to go extinct (~100% confidence). Causing human extinction would be criminal if done knowingly, so this satisfies Zach’s claim as written.
I’m leaving this light on links, because I’m not sure what of the above you might object to. I realize that you had many other objections to Zach’s framing, but I thought this could be something to drill into.
Edit: I’m not offering any money to respond, and independently of that, it’s 100% fine if you don’t want to respond.
I mostly take issue with the phrases “criminally insane”, “causing enormous damage”, and “he, almost uniquely, was not” connoting a more unusual and more actionable view than Eliezer (or almost anyone else) actually holds or would agree with.
Lots of people in the world are doing things that are straightforwardly non-optimal, often not even in their own narrow self-interest. This is mostly just the mistake side of conflict vs. mistake theory though, which seems relatively uncontroversial, at least on LW.
Eliezer has pointed out some of those mistakes in the context of AGI and other areas, but so have many others (Scott Alexander, Zack himself, etc.), in a variety of areas (housing policy, education policy, economics, etc.). Such explanations often come (implicitly or explicitly) with a call for others to change their behavior if they accept such arguments, but Eliezer doesn’t seem particularly strident in making such calls, compared to e.g. ordinary politicians, public policy advocates, or other rationalists.
Note, I’m not claiming that Eliezer does not hold some object-level views considered weird or extreme by most, e.g. that sufficiently intelligent AGI could take over the world, or that establishing multinational agreements for control and use of GPUs would be good policy.
But I agree with most or all those views because they seem correct on the merits, not because Eliezer (or anyone else) happened to say them. Eliezer may have been the one to point them out, for which he deserves credit, but he’s always explained his reasoning (sometimes at extreme length) and never to my knowledge asked anyone to just trust him about something like that and start making drastic behavioral changes as a result.
Again, leaving aside whether Eliezer himself has actually claimed this or would agree with the sentiment, it seems straightforwardly true to me that a world where the median IQ was 140 would look a lot different (and be a lot better off) than the current Earth. Whether or not it would look exactly like Eliezer’s speculative fictional world, it seems strange and uncharitable to me to characterize that view (that the world would be much better off with more intelligence) as extreme, or to interpret it as a demand or call to action for anything in particular.
I would summarize this as saying:
Zach (et al) are exaggerating how unusual/extreme/weird Yudkowsky’s positions are.
Zach (et al) are exaggerating how much Yudkowsky’s writings are an explicit call to action.
To the extent that Yudkowsky has unusual positions and calls for actions, you think he’s mostly correct on the merits.
Of these, I’d like to push on (1) a bit. However, I think this would probably work better as a new top-level post (working title “Yudkowsky on Yudkowsky”). To give a flavor, though, and because I’m quite likely to fail to write the top-level post, here’s an example. Shah and Yudkowsky on alignment failures.
I encourage you to follow the link to the rest of the conversation, which relates this to alignment work. So we have this phenomenon where one factor in humanity going extinct is that people don’t listen enough to Yudkowsky and his almost unique ability to speak concretely. This also supports (2) above—this isn’t an explicit call to action, he’s just observing a phenomenon.
A (1) take here is that the quote is cherry-picked, a joke, or an outlier, and his overall work implies a more modest self-assessment. A (3) take is that he really is almost uniquely able to speak concretely. My take (4) is that his self-assessment is positively biased. I interpret Zack’s “break-up” with Yudkowsky in the opening post as moving from a (3) model to a (4) model, and encouraging others to do the same.