My previous criticism was aimed at another post of yours, it likely wasn’t your main thesis. Some nitpicks I have with it are:
“Developing AGI responsibly requires massive safeguards that reduce performance, making AI less competitive” you could use the same argument for AIs which are “politically correct”, but we still choose to take this step, censorsing AIs and harming their performance, thus, it’s not impossible for us to make such choices as long as the social pressure is sufficiently high.
“The most reckless companies will outperform the most responsible ones” True in some ways, but most large companies are not all that reckless at all, which is why we are seeing many sequels, remakes, and clones in the entertainment sector. It’s also important to note that these incentives have been true for all of human nature, but that they’ve never mainfested very strongly until recent times. This suggests that that the antidote to Moloch is humanity itself, good faith, good taste and morality, and that these can beat game theoritical problem which are impossible when human beings are purely rational (i.e. inhuman).
We’re also assuming that AI becomes useful enough for us to disregard safety, i.e. that AI provides a lot of potential power. So far, this has not been true. AIs do not beat humans, companies are forcing LLMs into products but users did not ask for them. LLMs seem impressive at first, but after you get past the surface you realize that they’re somewhat incompetent. Governments won’t be playing around with human lives before these AIs provide large enough advantages.
“The moment an AGI can self-improve, it will begin optimizing its own intelligence.” This assumption is interesting, what does “intelligence” mean here? Many seems to just give these LLMS more knowledge and then call them more intelligent, but intelligence and knowledge are different things. Most “improvements” seem to lead to higher efficiency, but that’s just them being dumb faster or for cheaper. That said, self-improving intelligence is a dangerous concept.
I have many small objections like this to different parts of the essay, and they do add up, or at least add additional paths to how this could unfold.
I don’t think AIs will destroy humanity anytime soon (say, within 40 years). I do think that human extinction is possible, but I think it will be due to other things (like the low birthrate and its economic consequences. Also tech. Tech destroys the world for the same reasons that AIs do, it’s just slower).
I think it’s best to enjoy the years we have left instead of becoming depressed. I see a lot of people like you torturing themselves with x-risk problems (some people have killed themselves over Roko’s basilisk as well). Why not spend time with friends and loved ones?
Extra note: There’s no need to tie your identity together with your thesis. I’m the same kind of autistic as you. The futures I envision aren’t much better than yours, they’re just slightly different, so this is not some psychological cope. People misunderstand me as well, and 70% of the comments I leave across the internet get no engagement at all, not even negative feedback. But it’s alright. We can just see problems approaching many years before they’re visible to others.
My previous criticism was aimed at another post of yours, it likely wasn’t your main thesis. Some nitpicks I have with it are:
“Developing AGI responsibly requires massive safeguards that reduce performance, making AI less competitive” you could use the same argument for AIs which are “politically correct”, but we still choose to take this step, censorsing AIs and harming their performance, thus, it’s not impossible for us to make such choices as long as the social pressure is sufficiently high.
“The most reckless companies will outperform the most responsible ones” True in some ways, but most large companies are not all that reckless at all, which is why we are seeing many sequels, remakes, and clones in the entertainment sector. It’s also important to note that these incentives have been true for all of human nature, but that they’ve never mainfested very strongly until recent times. This suggests that that the antidote to Moloch is humanity itself, good faith, good taste and morality, and that these can beat game theoritical problem which are impossible when human beings are purely rational (i.e. inhuman).
We’re also assuming that AI becomes useful enough for us to disregard safety, i.e. that AI provides a lot of potential power. So far, this has not been true. AIs do not beat humans, companies are forcing LLMs into products but users did not ask for them. LLMs seem impressive at first, but after you get past the surface you realize that they’re somewhat incompetent. Governments won’t be playing around with human lives before these AIs provide large enough advantages.
“The moment an AGI can self-improve, it will begin optimizing its own intelligence.”
This assumption is interesting, what does “intelligence” mean here? Many seems to just give these LLMS more knowledge and then call them more intelligent, but intelligence and knowledge are different things. Most “improvements” seem to lead to higher efficiency, but that’s just them being dumb faster or for cheaper. That said, self-improving intelligence is a dangerous concept.
I have many small objections like this to different parts of the essay, and they do add up, or at least add additional paths to how this could unfold.
I don’t think AIs will destroy humanity anytime soon (say, within 40 years). I do think that human extinction is possible, but I think it will be due to other things (like the low birthrate and its economic consequences. Also tech. Tech destroys the world for the same reasons that AIs do, it’s just slower).
I think it’s best to enjoy the years we have left instead of becoming depressed. I see a lot of people like you torturing themselves with x-risk problems (some people have killed themselves over Roko’s basilisk as well). Why not spend time with friends and loved ones?
Extra note: There’s no need to tie your identity together with your thesis. I’m the same kind of autistic as you. The futures I envision aren’t much better than yours, they’re just slightly different, so this is not some psychological cope. People misunderstand me as well, and 70% of the comments I leave across the internet get no engagement at all, not even negative feedback. But it’s alright. We can just see problems approaching many years before they’re visible to others.