I’ve had lengthy one-on-one conversations with a few people (5, IIRC) since the Hugging Face incident about AI alignment (all of whom were either unaware of or unconcerned by the issue), and only one resulted in the person I was talking to being convinced that alignment is difficult, misalignment in highly intelligent AI is dangerous, and something ought to be done about it.
All other conversations had us moving back and forth between a number of different objections to misalignment risk, but in every case we eventually concluded that the source of our disagreement was that they did not believe that ASI was likely to ever exist. That belief, I think, meant that the possibility of agents autonomously developing goals in any meaningful sense was hard to believe, as well as the possibility that they might ever want or be able to deliberately circumvent guardrails against ‘bad’ behavior.I’m certainly not the first to notice this, but it was a striking thing for me to observe anyway, because it took getting through a number of seemingly unrelated topics to get to that conclusion.
I’m in the earth sciences, so a natural connection for me is to my conversations about climate change. Even though most people in my life believe that it is happening and nominally support action against it, I have met somewhat fewer people that have stated support for policies that would reduce greenhouse gas emissions if they also hurt the economy or would require meaningful lifestyle changes (e.g. drastically reducing the number of livestock raised for meat production). It seems that either some of these people are not as convinced about the likelihood or magnitude of harm as they say, or conviction in the likelihood of harm is insufficient reason to prioritize solutions to it over other things they value. My guess is that some combination of the two of these underlies a lot of collective complacency toward it.
I say this to express my concern that is is unlikely that the general public will be convinced to support a pause on capabilities research because of xrisk. They might support it for some other reason (jobs, datacenters, etc.), but the existential risk of misaligned ASI seems diffuse enough to give policies like this some substantial inertia. Until we have a particularly egregious example of a mustache-twirling rogue agent evading guardrails and cybersecurity defenders and committing some close-to-home harmful action, it will be easy to rationalize why we don’t need to do anything “drastic” about it, and (though this is just my hunch) people are and will continue to be motivated to find those rationalizations.
This is mostly anecdotal, so I might be underestimating people’s general willingness to update their beliefs.
I’ve had lengthy one-on-one conversations with a few people (5, IIRC) since the Hugging Face incident about AI alignment (all of whom were either unaware of or unconcerned by the issue), and only one resulted in the person I was talking to being convinced that alignment is difficult, misalignment in highly intelligent AI is dangerous, and something ought to be done about it.
All other conversations had us moving back and forth between a number of different objections to misalignment risk, but in every case we eventually concluded that the source of our disagreement was that they did not believe that ASI was likely to ever exist. That belief, I think, meant that the possibility of agents autonomously developing goals in any meaningful sense was hard to believe, as well as the possibility that they might ever want or be able to deliberately circumvent guardrails against ‘bad’ behavior. I’m certainly not the first to notice this, but it was a striking thing for me to observe anyway, because it took getting through a number of seemingly unrelated topics to get to that conclusion.
I’m in the earth sciences, so a natural connection for me is to my conversations about climate change. Even though most people in my life believe that it is happening and nominally support action against it, I have met somewhat fewer people that have stated support for policies that would reduce greenhouse gas emissions if they also hurt the economy or would require meaningful lifestyle changes (e.g. drastically reducing the number of livestock raised for meat production). It seems that either some of these people are not as convinced about the likelihood or magnitude of harm as they say, or conviction in the likelihood of harm is insufficient reason to prioritize solutions to it over other things they value. My guess is that some combination of the two of these underlies a lot of collective complacency toward it.
I say this to express my concern that is is unlikely that the general public will be convinced to support a pause on capabilities research because of xrisk. They might support it for some other reason (jobs, datacenters, etc.), but the existential risk of misaligned ASI seems diffuse enough to give policies like this some substantial inertia. Until we have a particularly egregious example of a mustache-twirling rogue agent evading guardrails and cybersecurity defenders and committing some close-to-home harmful action, it will be easy to rationalize why we don’t need to do anything “drastic” about it, and (though this is just my hunch) people are and will continue to be motivated to find those rationalizations.
This is mostly anecdotal, so I might be underestimating people’s general willingness to update their beliefs.