their primary concern seems to be driven by the nature of existing deep learning technologies. This is suggested by their use of the phrase “anything remotely like current techniques”, which suggests that their core worries stem largely from deep learning rather than all potential AI development pathways
You know better! Eliezer at least has been arguing these points far before DL!
He has been warning of a significant risk of catastrophe for a long time, but unless I’m mistaken, he only began explicitly and primarily arguing for a high probability of catastrophe more recently, around the time deep learning emerged. This distinction is essential to my argument, and was highlighted explicitly by my comment.
Yes, I agree your whole comment sucks. I know you know there is a difference between p(doom) and p(doom|AGI soon), and your reasons for having a high p(doom | AGI soon) and low p(doom) can be very different. Indeed a whole factor of p(AGI soon) different!
So we can get the observed shift with most of the “highly technical DL-specific considerations” mainly updating the p(AGI soon) factor via the incredibly complicated and arcane practice of… extrapolating benchmark scores.
Indeed, the fact AGI seems to be arriving so quickly is the main reason most people are worried!
This is not to say they like deep learning. There can be additional reasons deep learning is bad in their book, but is deep learning a core part of their arguments? Hell no! Do you know how I know? I’ve actually read them! Indeed, if you type site:arbital.greaterwrong.com “deep learning” into google, you get back two results. Compare with site:arbital.greaterwrong.com “utility function”, which gives you 5 pages. Now which do you think is more central to their high p(doom | AGI in 5 years)?
I wasn’t asking for your evaluation of the rest of my comment. I was clarifying a specific point because it seemed you had misunderstood what I was saying.
So we can get the observed shift with most of the “highly technical DL-specific considerations” mainly updating the p(AGI soon) factor via the incredibly complicated and arcane practice of… extrapolating benchmark scores.
Indeed, the fact AGI seems to be arriving so quickly is the main reason most people are worried!
If someone says their high p(doom) is driven by short timelines, what they likely mean is that AGI is now expected to arrive via a certain method—namely, deep learning—that is perceived as riskier than what might have emerged under slower or more deliberate development. If that’s the case, it directly supports my core point.
This explanation makes sense to me since expecting AGI to arrive soon doesn’t by itself justify a high probability of doom. After all, it would have been reasonable to have always believed AGI would come eventually, and it would have been unjustified to increase one’s p(doom) over time merely because time is passing.
There can be additional reasons deep learning is bad in their book, but is deep learning a core part of their arguments? Hell no! Do you know how I know? I’ve actually read them!
I think you’re conflating two distinct issues: first, what initially made people worry about AI risk at all; and second, what made people think doom is likely as opposed to merely a possibility worth taking seriously. I’m addressing the second point, not the first.
Please try to engage with what I’m actually saying, rather than continuing to misrepresent my position.
Please try to engage with what I’m actually saying, rather than continuing to misrepresent my position.
It seems everyone has this problem with your writing, have you considered speaking more clearly or perhaps considering people understand you fully and it is you who are wrong?
In this case, I believe it’s the latter, since
If someone says their high p(doom) is driven by short timelines, what they likely mean is that AGI is now expected to arrive via a certain method—namely, deep learning—that is perceived as riskier than what might have emerged under slower or more deliberate development. If that’s the case, it directly supports my core point.
Really? I thought your core point was
It’s perhaps worth highlighting the significant tension between two contrasting claims: on the one hand, the idea that modern AI doomerism was “anticipated” as early as the 19th century, and on the other, the idea that modern AI doom arguments are rationally grounded in a technical understanding of today’s deep learning systems.
In which case I did explain why there is no tension, as can be seen from my saying
So we can get the observed shift with most of the “highly technical DL-specific considerations” mainly updating the p(AGI soon) factor via the incredibly complicated and arcane practice of… extrapolating benchmark scores.
That is, it is a very strange thing to say there is a “significant tension” between having high p(doom | AGI soon) on first principles reasoning, and to have p(AGI soon) get updated by benchmark scores.
Yes — Garrett Baker repeatedly and materially misrepresents what Matthew is saying.
I have custom instructions turned off, and I haven’t turned on the memory feature, so there’s no strong reason to expect it to behave sycophantically (that I’m aware of). And o3 said it doesn’t know which side I’m on. I expect most other LLMs will say something similar when given neutral prompts and the full context.
(Not that this is strong evidence. But I think it undermines your claim by at least a bit.)
o3 has the same conclusion with a slightly different prompt.
Read this comment exchange and come to a definitive conclusion about whether Garrett Baker is accurately representing Matthew. Focus on content rather than tone:
Conclusion: Garrett is not accurately representing Matthew’s position. Below is a point‑by‑point comparison that shows where Garrett’s paraphrases diverge from what Matthew is actually claiming (ignoring tone and focusing only on the content).
It seems everyone has this problem with your writing, have you considered speaking more clearly or perhaps considering people understand you fully and it is you who are wrong?
I reject the premise. In general, my writing is interpreted significantly more accurately when I’m not signaling skepticism about AI risk on LessWrong. For most other topics, including on this site, readers tend to understand my points reasonably well, especially when the subject is less controversial.
This could perhaps mean I’m uniquely unclear when discussing AI risk. It’s also very plausible that the topic itself is unusually prone to misrepresentation. Still, I think a major factor is that people are often uncharitable toward unpopular viewpoints they strongly disagree with, which accounts for much of the pushback I receive on this subject.
Specifically, the idea is that AI going well for humans would require a detailed theory of how to encode human values in form suitable for machine optimization, and the relevance of deep learning is that Yudkowsky and Soares think that deep learning is on track to provide the superhuman optimization without the theory of values. You’re correct to note that this is a stance according to which “artificial life is by default bad, dangerous, or disvaluable,” but I think the way you contrast it with the claim that “biological life is by default good or preferable” is getting the nuances slightly wrong: independently-evolved biological aliens with superior intelligence would also be dangerous for broadly similar reasons.
Didn’t you have a post where you argued that it’s a consequence of their view that biological aliens are better, morally speaking, than artificial earth originating life, or did I misunderstand?
You know better! Eliezer at least has been arguing these points far before DL!
He has been warning of a significant risk of catastrophe for a long time, but unless I’m mistaken, he only began explicitly and primarily arguing for a high probability of catastrophe more recently, around the time deep learning emerged. This distinction is essential to my argument, and was highlighted explicitly by my comment.
Yes, I agree your whole comment sucks. I know you know there is a difference between p(doom) and p(doom|AGI soon), and your reasons for having a high p(doom | AGI soon) and low p(doom) can be very different. Indeed a whole factor of p(AGI soon) different!
So we can get the observed shift with most of the “highly technical DL-specific considerations” mainly updating the p(AGI soon) factor via the incredibly complicated and arcane practice of… extrapolating benchmark scores.
Indeed, the fact AGI seems to be arriving so quickly is the main reason most people are worried!
This is not to say they like deep learning. There can be additional reasons deep learning is bad in their book, but is deep learning a core part of their arguments? Hell no! Do you know how I know? I’ve actually read them! Indeed, if you type
site:arbital.greaterwrong.com “deep learning”into google, you get back two results. Compare withsite:arbital.greaterwrong.com “utility function”, which gives you 5 pages. Now which do you think is more central to their high p(doom | AGI in 5 years)?I wasn’t asking for your evaluation of the rest of my comment. I was clarifying a specific point because it seemed you had misunderstood what I was saying.
If someone says their high p(doom) is driven by short timelines, what they likely mean is that AGI is now expected to arrive via a certain method—namely, deep learning—that is perceived as riskier than what might have emerged under slower or more deliberate development. If that’s the case, it directly supports my core point.
This explanation makes sense to me since expecting AGI to arrive soon doesn’t by itself justify a high probability of doom. After all, it would have been reasonable to have always believed AGI would come eventually, and it would have been unjustified to increase one’s p(doom) over time merely because time is passing.
I think you’re conflating two distinct issues: first, what initially made people worry about AI risk at all; and second, what made people think doom is likely as opposed to merely a possibility worth taking seriously. I’m addressing the second point, not the first.
Please try to engage with what I’m actually saying, rather than continuing to misrepresent my position.
It seems everyone has this problem with your writing, have you considered speaking more clearly or perhaps considering people understand you fully and it is you who are wrong?
In this case, I believe it’s the latter, since
Really? I thought your core point was
In which case I did explain why there is no tension, as can be seen from my saying
That is, it is a very strange thing to say there is a “significant tension” between having high p(doom | AGI soon) on first principles reasoning, and to have p(AGI soon) get updated by benchmark scores.
This is o3′s take, for what it’s worth:
I have custom instructions turned off, and I haven’t turned on the memory feature, so there’s no strong reason to expect it to behave sycophantically (that I’m aware of). And o3 said it doesn’t know which side I’m on. I expect most other LLMs will say something similar when given neutral prompts and the full context.
(Not that this is strong evidence. But I think it undermines your claim by at least a bit.)
o3 has the same conclusion with a slightly different prompt.
That link seems to be broken.ETA: now fixed by Thomas.oops, this was on my work account from which you can’t make public links. Replaced the link with the prompt and beginning of o3 output.
I get the same result with Claude, but when I push at all it caves & says I understand you fine.
It seems a crux is what you mean by “tension”.
Just to offer my two cents, I do not have this problem and I think Matthew is extremely clear.
Can you rephrase his argument in your own words? In particular, define what “tension” means.
I reject the premise. In general, my writing is interpreted significantly more accurately when I’m not signaling skepticism about AI risk on LessWrong. For most other topics, including on this site, readers tend to understand my points reasonably well, especially when the subject is less controversial.
This could perhaps mean I’m uniquely unclear when discussing AI risk. It’s also very plausible that the topic itself is unusually prone to misrepresentation. Still, I think a major factor is that people are often uncharitable toward unpopular viewpoints they strongly disagree with, which accounts for much of the pushback I receive on this subject.
Specifically, the idea is that AI going well for humans would require a detailed theory of how to encode human values in form suitable for machine optimization, and the relevance of deep learning is that Yudkowsky and Soares think that deep learning is on track to provide the superhuman optimization without the theory of values. You’re correct to note that this is a stance according to which “artificial life is by default bad, dangerous, or disvaluable,” but I think the way you contrast it with the claim that “biological life is by default good or preferable” is getting the nuances slightly wrong: independently-evolved biological aliens with superior intelligence would also be dangerous for broadly similar reasons.
Didn’t you have a post where you argued that it’s a consequence of their view that biological aliens are better, morally speaking, than artificial earth originating life, or did I misunderstand?