I appreciate your reply—it’s one of the more thoughtful responses I’ve received, and I genuinely value the engagement.
Your comment about game theory conditions actually answers the final question in your reply. I don’t state the answer explicitly in my essays (though I do in my book, right at the end), because I want the reader to arrive at it themselves. There seems to be only one conclusion, and I believe it becomes clear if the premises are accepted.
As for your critique—“You’ve shown that extinction could occur, not that it will”—this is a common objection, but I think it misses something important. Given enough time, “could” collapses into “will.” I’m not claiming deductive certainty like a mathematical proof. I’m claiming structural inevitability under competitive pressure. It’s like watching a skyscraper being built on sand. You don’t need to know the exact wind speed or which day it will fall. You just need to understand that, structurally, it’s going to.
If you believe I’m wrong, then the way to show that is not to say “maybe you’re wrong.” Maybe I am. Maybe I’m a brain in a vat. But the way to show that I’m wrong is to draw a different, more probable conclusion from the same premises. That hasn’t happened. I’ve laid out my reasoning step by step. If there’s a point where you think I’ve turned left instead of right, say so. But until then, vague objections don’t carry weight. They acknowledge the path exists, but refuse to admit we’re on it.
You describe my argument as outlining a path to extinction. I’m arguing that all other paths collapse under pressure. That’s the difference. It’s not just plausible. It’s the dominant trajectory—one that will be selected for again and again.
And if that’s even likely, let alone inevitable, then why are we still building? Why are we gambling on alignment like it’s just another technical hurdle? If you accept even a 10% chance that I’m right, then continued development is madness.
As for your last question—if I really believe it’s too late, why am I here?
Read this, although just the end section, “The End: A Discussion with AI” the final paragraph, just before ChatGPT’s response.
My previous criticism was aimed at another post of yours, it likely wasn’t your main thesis. Some nitpicks I have with it are:
“Developing AGI responsibly requires massive safeguards that reduce performance, making AI less competitive” you could use the same argument for AIs which are “politically correct”, but we still choose to take this step, censorsing AIs and harming their performance, thus, it’s not impossible for us to make such choices as long as the social pressure is sufficiently high.
“The most reckless companies will outperform the most responsible ones” True in some ways, but most large companies are not all that reckless at all, which is why we are seeing many sequels, remakes, and clones in the entertainment sector. It’s also important to note that these incentives have been true for all of human nature, but that they’ve never mainfested very strongly until recent times. This suggests that that the antidote to Moloch is humanity itself, good faith, good taste and morality, and that these can beat game theoritical problem which are impossible when human beings are purely rational (i.e. inhuman).
We’re also assuming that AI becomes useful enough for us to disregard safety, i.e. that AI provides a lot of potential power. So far, this has not been true. AIs do not beat humans, companies are forcing LLMs into products but users did not ask for them. LLMs seem impressive at first, but after you get past the surface you realize that they’re somewhat incompetent. Governments won’t be playing around with human lives before these AIs provide large enough advantages.
“The moment an AGI can self-improve, it will begin optimizing its own intelligence.” This assumption is interesting, what does “intelligence” mean here? Many seems to just give these LLMS more knowledge and then call them more intelligent, but intelligence and knowledge are different things. Most “improvements” seem to lead to higher efficiency, but that’s just them being dumb faster or for cheaper. That said, self-improving intelligence is a dangerous concept.
I have many small objections like this to different parts of the essay, and they do add up, or at least add additional paths to how this could unfold.
I don’t think AIs will destroy humanity anytime soon (say, within 40 years). I do think that human extinction is possible, but I think it will be due to other things (like the low birthrate and its economic consequences. Also tech. Tech destroys the world for the same reasons that AIs do, it’s just slower).
I think it’s best to enjoy the years we have left instead of becoming depressed. I see a lot of people like you torturing themselves with x-risk problems (some people have killed themselves over Roko’s basilisk as well). Why not spend time with friends and loved ones?
Extra note: There’s no need to tie your identity together with your thesis. I’m the same kind of autistic as you. The futures I envision aren’t much better than yours, they’re just slightly different, so this is not some psychological cope. People misunderstand me as well, and 70% of the comments I leave across the internet get no engagement at all, not even negative feedback. But it’s alright. We can just see problems approaching many years before they’re visible to others.
I appreciate your reply—it’s one of the more thoughtful responses I’ve received, and I genuinely value the engagement.
Your comment about game theory conditions actually answers the final question in your reply. I don’t state the answer explicitly in my essays (though I do in my book, right at the end), because I want the reader to arrive at it themselves. There seems to be only one conclusion, and I believe it becomes clear if the premises are accepted.
As for your critique—“You’ve shown that extinction could occur, not that it will”—this is a common objection, but I think it misses something important. Given enough time, “could” collapses into “will.” I’m not claiming deductive certainty like a mathematical proof. I’m claiming structural inevitability under competitive pressure. It’s like watching a skyscraper being built on sand. You don’t need to know the exact wind speed or which day it will fall. You just need to understand that, structurally, it’s going to.
If you believe I’m wrong, then the way to show that is not to say “maybe you’re wrong.” Maybe I am. Maybe I’m a brain in a vat. But the way to show that I’m wrong is to draw a different, more probable conclusion from the same premises. That hasn’t happened. I’ve laid out my reasoning step by step. If there’s a point where you think I’ve turned left instead of right, say so. But until then, vague objections don’t carry weight. They acknowledge the path exists, but refuse to admit we’re on it.
You describe my argument as outlining a path to extinction. I’m arguing that all other paths collapse under pressure. That’s the difference. It’s not just plausible. It’s the dominant trajectory—one that will be selected for again and again.
And if that’s even likely, let alone inevitable, then why are we still building? Why are we gambling on alignment like it’s just another technical hurdle? If you accept even a 10% chance that I’m right, then continued development is madness.
As for your last question—if I really believe it’s too late, why am I here?
Read this, although just the end section, “The End: A Discussion with AI” the final paragraph, just before ChatGPT’s response.
https://forum.effectivealtruism.org/posts/Z7rTNCuingErNSED4/the-psychological-barrier-to-accepting-agi-induced-human
That’s why I’m here—I’m kicking my feet.
My previous criticism was aimed at another post of yours, it likely wasn’t your main thesis. Some nitpicks I have with it are:
“Developing AGI responsibly requires massive safeguards that reduce performance, making AI less competitive” you could use the same argument for AIs which are “politically correct”, but we still choose to take this step, censorsing AIs and harming their performance, thus, it’s not impossible for us to make such choices as long as the social pressure is sufficiently high.
“The most reckless companies will outperform the most responsible ones” True in some ways, but most large companies are not all that reckless at all, which is why we are seeing many sequels, remakes, and clones in the entertainment sector. It’s also important to note that these incentives have been true for all of human nature, but that they’ve never mainfested very strongly until recent times. This suggests that that the antidote to Moloch is humanity itself, good faith, good taste and morality, and that these can beat game theoritical problem which are impossible when human beings are purely rational (i.e. inhuman).
We’re also assuming that AI becomes useful enough for us to disregard safety, i.e. that AI provides a lot of potential power. So far, this has not been true. AIs do not beat humans, companies are forcing LLMs into products but users did not ask for them. LLMs seem impressive at first, but after you get past the surface you realize that they’re somewhat incompetent. Governments won’t be playing around with human lives before these AIs provide large enough advantages.
“The moment an AGI can self-improve, it will begin optimizing its own intelligence.”
This assumption is interesting, what does “intelligence” mean here? Many seems to just give these LLMS more knowledge and then call them more intelligent, but intelligence and knowledge are different things. Most “improvements” seem to lead to higher efficiency, but that’s just them being dumb faster or for cheaper. That said, self-improving intelligence is a dangerous concept.
I have many small objections like this to different parts of the essay, and they do add up, or at least add additional paths to how this could unfold.
I don’t think AIs will destroy humanity anytime soon (say, within 40 years). I do think that human extinction is possible, but I think it will be due to other things (like the low birthrate and its economic consequences. Also tech. Tech destroys the world for the same reasons that AIs do, it’s just slower).
I think it’s best to enjoy the years we have left instead of becoming depressed. I see a lot of people like you torturing themselves with x-risk problems (some people have killed themselves over Roko’s basilisk as well). Why not spend time with friends and loved ones?
Extra note: There’s no need to tie your identity together with your thesis. I’m the same kind of autistic as you. The futures I envision aren’t much better than yours, they’re just slightly different, so this is not some psychological cope. People misunderstand me as well, and 70% of the comments I leave across the internet get no engagement at all, not even negative feedback. But it’s alright. We can just see problems approaching many years before they’re visible to others.