But I overall think working on alignment is largely more urgent. Being able to understand what’s going on at all inside a neural net, and advocating that companies be required to understand what’s going on before developing new/bigger/better models, seems like a convergent goal relevant to both human extinction and astronomical suffering.
There’s also something of a Maslow Hierarchy of needs thing where I think getting to “okay, we’re not all dead and we have some control over the future” seems more like the next major step to focus on, in part since it requires fewer philosophical assumptions.
But I am pretty in favor of people a) figuring out how to build AI evals that check for sentience, and/or finding a reasonable bright line that’s like “if your AI can do X, there’s at least a sizable chance that it’s sentient”, and b) advocating companies or governments halt training runs or deployment if that bright line is crossed.
Note, like, Eliezer did go out of his way to mention AI consciousness as a problem in his TIME article and in some tweets.
But I overall think working on alignment is largely more urgent. Being able to understand what’s going on at all inside a neural net, and advocating that companies be required to understand what’s going on before developing new/bigger/better models, seems like a convergent goal relevant to both human extinction and astronomical suffering.
Fwiw, Lukas’s comment link to a post arguing against that and I 100% agree with it. I think the “Alignment will solve s-risks as well anyway” is one the most untrue and harmful widespread memes in the EA/LW community.
Nod (fyi I vaguely remembered that comment but couldn’t find it a second time while I was writing my own answer)
I do think “AI targeted at optimizing a good goal” is more likely to near miss if precautions aren’t taken and I do think that’s quite important. I did carefully not say “alignment automatically solves s-risks”, I said it was a convergent goal that seemed more important to me overall. I do think that’s a reasonable thing to disagree on.
I suppose my shooting range metaphor falls short here. Maybe alignment is like teaching a kid to be an ace race car driver, and s-risks are accidents on normal roads. There it also depends on the details whether the ace race car driver will drive safely on normal roads.
Note, something in your framing here seems focused on psychologizing in a way that feels unnecessary and unhelpful, although I’m not sure whether I particularly object to it. [Specifically, the part where your question title is “why are we complacent?” rather than “why aren’t more people taking Particular Actions or prioritizing AI Hell above AI extinction”]
Good point. I can still change it. What title would you vote for? I spent a lot of time vacillating between titles and don’t have a strong opinion. These were the options that I considered:
Why not s-risks? A poll.
Why are we so complacent about AI hell?
Why aren’t we taking s-risks from AI more seriously?
Why do so few people care about s-risks from AI?
Why are we ignoring the risks of AI hell?
What’s holding us back from addressing s-risks from AI?
Why aren’t we doing more to prevent s-risks from AI?
What will it take to get people to care about s-risks from AI?
“Why aren’t more people prioritizing work on S-risks more heavily” seems better to me and seems like the question you probably actually care about. Question-titles that are making (in many cases inaccurate) claims about people’s motivations seem more fraught and unhelpfully opinionated.
Oh, true! Digital sentience is also an important point! A bit of an intuition pump is that if you consider a certain animal to be sentient (at least with some probability), then an em of that animal’s brain may be sentient with a similar probability. If an AI is powerful enough to run such ems, the question is no longer whether digital sentience is possible but why an AI would run such an em.
The Maslow hierarchy is reverse for me, i.e. rather dead/disempowered than being tortured, but that’s just a personal thing. In the end it’s more important what the acausal moral compromise says, I think.
Yeah to be clear my mainline prediction is that an unfriendly AI goes through some period of simulating lots of humans (less likely to simulate animals IMO) as part of it’s strategizing process, kills humanity, and then goes on to do mostly non-sentient things.
There might be a second phase where it does some kind of weird acausal thing, not sure.
I don’t know that in my mainline prediction the simulation process results in much more negative utility than the extinction part. I think the AI probably has to do much of it’s strategizing without enough compute to simulate vast numbers of humans, and I weakly bet against those simulations ending up suffering in a way that ends up outweighing human extinction.
There are other moderately likely worlds IMO and yeah I think s-risk is a pretty real concern.
Speaking for myself: I care a bunch about s-risks. I listed “the AI might simulate sentient beings” as a failure mode in “Carefully Bootstrapped Alignment” is organizationally hard.
But I overall think working on alignment is largely more urgent. Being able to understand what’s going on at all inside a neural net, and advocating that companies be required to understand what’s going on before developing new/bigger/better models, seems like a convergent goal relevant to both human extinction and astronomical suffering.
There’s also something of a Maslow Hierarchy of needs thing where I think getting to “okay, we’re not all dead and we have some control over the future” seems more like the next major step to focus on, in part since it requires fewer philosophical assumptions.
But I am pretty in favor of people a) figuring out how to build AI evals that check for sentience, and/or finding a reasonable bright line that’s like “if your AI can do X, there’s at least a sizable chance that it’s sentient”, and b) advocating companies or governments halt training runs or deployment if that bright line is crossed.
Note, like, Eliezer did go out of his way to mention AI consciousness as a problem in his TIME article and in some tweets.
Fwiw, Lukas’s comment link to a post arguing against that and I 100% agree with it. I think the “Alignment will solve s-risks as well anyway” is one the most untrue and harmful widespread memes in the EA/LW community.
Nod (fyi I vaguely remembered that comment but couldn’t find it a second time while I was writing my own answer)
I do think “AI targeted at optimizing a good goal” is more likely to near miss if precautions aren’t taken and I do think that’s quite important. I did carefully not say “alignment automatically solves s-risks”, I said it was a convergent goal that seemed more important to me overall. I do think that’s a reasonable thing to disagree on.
I suppose my shooting range metaphor falls short here. Maybe alignment is like teaching a kid to be an ace race car driver, and s-risks are accidents on normal roads. There it also depends on the details whether the ace race car driver will drive safely on normal roads.
Note, something in your framing here seems focused on psychologizing in a way that feels unnecessary and unhelpful, although I’m not sure whether I particularly object to it. [Specifically, the part where your question title is “why are we complacent?” rather than “why aren’t more people taking Particular Actions or prioritizing AI Hell above AI extinction”]
Good point. I can still change it. What title would you vote for? I spent a lot of time vacillating between titles and don’t have a strong opinion. These were the options that I considered:
Why not s-risks? A poll.
Why are we so complacent about AI hell?
Why aren’t we taking s-risks from AI more seriously?
Why do so few people care about s-risks from AI?
Why are we ignoring the risks of AI hell?
What’s holding us back from addressing s-risks from AI?
Why aren’t we doing more to prevent s-risks from AI?
What will it take to get people to care about s-risks from AI?
“Why aren’t more people prioritizing work on S-risks more heavily” seems better to me and seems like the question you probably actually care about. Question-titles that are making (in many cases inaccurate) claims about people’s motivations seem more fraught and unhelpfully opinionated.
Thx! I’ll probably drop the “more heavily” for stylistic reasons, but otherwise that sounds good to me!
Oh, true! Digital sentience is also an important point! A bit of an intuition pump is that if you consider a certain animal to be sentient (at least with some probability), then an em of that animal’s brain may be sentient with a similar probability. If an AI is powerful enough to run such ems, the question is no longer whether digital sentience is possible but why an AI would run such an em.
The Maslow hierarchy is reverse for me, i.e. rather dead/disempowered than being tortured, but that’s just a personal thing. In the end it’s more important what the acausal moral compromise says, I think.
Yeah to be clear my mainline prediction is that an unfriendly AI goes through some period of simulating lots of humans (less likely to simulate animals IMO) as part of it’s strategizing process, kills humanity, and then goes on to do mostly non-sentient things.
There might be a second phase where it does some kind of weird acausal thing, not sure.
I don’t know that in my mainline prediction the simulation process results in much more negative utility than the extinction part. I think the AI probably has to do much of it’s strategizing without enough compute to simulate vast numbers of humans, and I weakly bet against those simulations ending up suffering in a way that ends up outweighing human extinction.
There are other moderately likely worlds IMO and yeah I think s-risk is a pretty real concern.