Thanks for the list, I’ve already read a lot of those posts, but I still remain unconvinced. Are you convinced by any of those arguments? Do you suggest I take a closer look to some posts?
But honestly, with the AI risk statement signed by so many prominent scientists and engineer, debating that AI risks somehow don’t exists seems to be just a fringe anti-climate-change-like opinion held by few stubborn people (or people just not properly introduced to the arguments). I find it funny that we are in a position where in the possible counter arguments appears “angels might save us”, thanks for the chuckle.
To be fair I think this post argues about how overconfident Yudkosky is at placing doom at 95%+, and sure, why not… But, as a person that doesn’t want to personally die, I cannot say that “it will be fine” unless I have good arguments as to why the p(doom) should be less than 0.1% and not “only 20%”!
But honestly, with the AI risk statement signed by so many prominent scientists and engineer,
Well, yes, the statement says “should be a global priority alongside other societal-scale risks”, not anything about brakes on capabilities research, or privileging this risk over others. This is not at all the cautionista stance. Not even watered down. It is only to raise the public profile of this particular x-risk existence.
I don’t get you. You are upset about people saying that we should scale back capabilities research, while at the same time holding the opinion that we are not doomed because we won’t get to ASI? You are worried that people might try to stop the technology that in your opinion may not happen?? The technology that if does indeed happen, you agree that “If [ASI] us wants us gone, we would be gone”?!?
Said this, maybe you are misunderstanding the people that are calling for a stop. I don’t think anyone is proposing to stop narrow AI capabilities. Just the dangerous kind of general intelligence “larger than GPT-4”. Self-driving cars good, automated general decision-making bad.
I’d also still like to hear your opinion on my counter arguments on the object level.
I did listen to that post, and while I don’t remember all the points, I do remember that it didn’t convince me that alignment is easy and, like Christiano’s post “Where I agree and disagree with Eliezer”, it just seems to be like “p(doom) of 95%+ plus is too much, it’s probably something like 10-50%” which is still incredibly unacceptably high to continue “business as usual”. I have faith that something will be done: regulation and breakthrough will happen, but it seems likely that it won’t be enough.
It comes down to safety mindset. There are very few and sketchy reasons to expect that by default an ASI will care about humans enough, so it not safe to build one until shown otherwise (preferably without actually creating one). And if I had to point out a single cause for my own high p(doom), it is the fact that we humans iterate all of our engineering to iron out all of the kinks, while with a technology that is itself adversarial, iteration might not be available (get it right the first time we deploy powerful AI).
Who do you think are the two or three smartest people to be skeptical of AI killing all humans? I think maybe Yann LeCunn and Andrew Ng.
Sure, those two. I don’t know about Ng (he recently had a private discussion with Hinton, but I don’t know what he thinks now), but I know LeCun hasn’t really engaged with the ideas and just relies on the concept that “it’s an extreme idea”. But as I said, having the position “AI doesn’t pose an existential threat” seems to be fringe nowadays.
If I dumb the argument down enough I get stuff like “intelligence/cognition/optimization is dangerous, and, whatever the reasons, we currently have zero reliable ideas on how to make a powerful general intelligence safe (eg. RLHF doesn’t work well enough as GPT-4 still lies/hallucinates and is jailbroken way too easily)” which is evidence based, not weird and not extreme.
Thanks for the list, I’ve already read a lot of those posts, but I still remain unconvinced. Are you convinced by any of those arguments? Do you suggest I take a closer look to some posts?
But honestly, with the AI risk statement signed by so many prominent scientists and engineer, debating that AI risks somehow don’t exists seems to be just a fringe anti-climate-change-like opinion held by few stubborn people (or people just not properly introduced to the arguments). I find it funny that we are in a position where in the possible counter arguments appears “angels might save us”, thanks for the chuckle.
To be fair I think this post argues about how overconfident Yudkosky is at placing doom at 95%+, and sure, why not… But, as a person that doesn’t want to personally die, I cannot say that “it will be fine” unless I have good arguments as to why the p(doom) should be less than 0.1% and not “only 20%”!
Well, yes, the statement says “should be a global priority alongside other societal-scale risks”, not anything about brakes on capabilities research, or privileging this risk over others. This is not at all the cautionista stance. Not even watered down. It is only to raise the public profile of this particular x-risk existence.
I don’t get you. You are upset about people saying that we should scale back capabilities research, while at the same time holding the opinion that we are not doomed because we won’t get to ASI? You are worried that people might try to stop the technology that in your opinion may not happen?? The technology that if does indeed happen, you agree that “If [ASI] us wants us gone, we would be gone”?!?
Said this, maybe you are misunderstanding the people that are calling for a stop. I don’t think anyone is proposing to stop narrow AI capabilities. Just the dangerous kind of general intelligence “larger than GPT-4”. Self-driving cars good, automated general decision-making bad.
I’d also still like to hear your opinion on my counter arguments on the object level.
Consider also reading Scott Aaronson (whose sabbatical at OpenAI is about to end):
https://scottaaronson.blog/?p=7266
https://scottaaronson.blog/?p=7230
https://scottaaronson.blog/?p=7174
https://scottaaronson.blog/?p=7064
I did listen to that post, and while I don’t remember all the points, I do remember that it didn’t convince me that alignment is easy and, like Christiano’s post “Where I agree and disagree with Eliezer”, it just seems to be like “p(doom) of 95%+ plus is too much, it’s probably something like 10-50%” which is still incredibly unacceptably high to continue “business as usual”. I have faith that something will be done: regulation and breakthrough will happen, but it seems likely that it won’t be enough.
It comes down to safety mindset. There are very few and sketchy reasons to expect that by default an ASI will care about humans enough, so it not safe to build one until shown otherwise (preferably without actually creating one). And if I had to point out a single cause for my own high p(doom), it is the fact that we humans iterate all of our engineering to iron out all of the kinks, while with a technology that is itself adversarial, iteration might not be available (get it right the first time we deploy powerful AI).
Sure, those two. I don’t know about Ng (he recently had a private discussion with Hinton, but I don’t know what he thinks now), but I know LeCun hasn’t really engaged with the ideas and just relies on the concept that “it’s an extreme idea”. But as I said, having the position “AI doesn’t pose an existential threat” seems to be fringe nowadays.
If I dumb the argument down enough I get stuff like “intelligence/cognition/optimization is dangerous, and, whatever the reasons, we currently have zero reliable ideas on how to make a powerful general intelligence safe (eg. RLHF doesn’t work well enough as GPT-4 still lies/hallucinates and is jailbroken way too easily)” which is evidence based, not weird and not extreme.