This is great work. To zero in on the part that repulsed me rather than gave me clear new ways of putting things:
“My colleague Harriet, a senior researcher in her subfield, left her tenured professorship to work at a frontier AI lab not too long ago. This summer, we got on the phone and chatted about AI risk.
I showed her a seed of this essay and explained that I wanted to create common knowledge about x-risk in the mathematical community. She was dismissive.
She said, “On the current trajectory, humanity is 99% doomed, and this is a whole lot of effort for something that won’t obviously help.””
Harriet ought to be screaming her head off! Rather than making the problem worse (as I strongly suspect she is in her position) and belittling you. Outrageous.
I think we should be more considerate of people as they grapple with the heft of the problem. Many will continue to keep their heads in the sand, some will swing wildly in the opposite direction because [we’re definitely doomed, there’s nothing to be done] can sometimes be the next psychologically safest place to be after [we’re definitely ok, there’s nothing to worry about].
Being empathetic and being nice aren’t always the same thing. Loving actions are sometimes adversarial. When I imagine myself in the same position, I would want a friend to confront me and demand to know what the hell I’m thinking.
“We are doomed and there is nothing to be done, so I might as well doom us harder” can be made psychologically unsafe. It’s plenty of reason to end a friendship. They should feel pressure and shame for doing such a taboo thing, not to punish them, but because the lack of hard pushback acts as tacit approval, as if we can all be friends at the end of the day, as if friendship is more important than the death of all things.
Is it? I have often felt my relationship with a friend strengthen when I frankly tell them that they ought to quit their jobs on ethical and/or mission grounds. (Prior to this whole AI labs business.)
Sorry, I should have expanded a bit more. The quoted part is only the first five minutes of a two-hour conversation; I definitely believe in having frank conversations about x-risk with people who are on the ground.
What I don’t believe in is jumping directly to [here’s what you ought to do], which feels deeply condescending, as opposed to staying at [let’s find our cruxes, why do you believe what you believe?].
It is also not obvious to me that having them leave is the right decision, as it might have been ~5 years ago. There seems to be delicate internal opinion politics right now in the labs where the x-risk-pilled are a non-negligible minority and their presence could significantly improve our chances of (for example) future pauses/bans, or deployment of alignment ideas, or alignment problems being solved by powerful internal models.
This is great work. To zero in on the part that repulsed me rather than gave me clear new ways of putting things:
“My colleague Harriet, a senior researcher in her subfield, left her tenured professorship to work at a frontier AI lab not too long ago. This summer, we got on the phone and chatted about AI risk.
I showed her a seed of this essay and explained that I wanted to create common knowledge about x-risk in the mathematical community. She was dismissive.
She said, “On the current trajectory, humanity is 99% doomed, and this is a whole lot of effort for something that won’t obviously help.””
Harriet ought to be screaming her head off! Rather than making the problem worse (as I strongly suspect she is in her position) and belittling you. Outrageous.
I think we should be more considerate of people as they grapple with the heft of the problem. Many will continue to keep their heads in the sand, some will swing wildly in the opposite direction because [we’re definitely doomed, there’s nothing to be done] can sometimes be the next psychologically safest place to be after [we’re definitely ok, there’s nothing to worry about].
Being empathetic and being nice aren’t always the same thing. Loving actions are sometimes adversarial. When I imagine myself in the same position, I would want a friend to confront me and demand to know what the hell I’m thinking.
“We are doomed and there is nothing to be done, so I might as well doom us harder” can be made psychologically unsafe. It’s plenty of reason to end a friendship. They should feel pressure and shame for doing such a taboo thing, not to punish them, but because the lack of hard pushback acts as tacit approval, as if we can all be friends at the end of the day, as if friendship is more important than the death of all things.
Pressuring the people at frontier ai labs who believe in x-risk to leave is an extremely naive way of relating to doom.
Is it? I have often felt my relationship with a friend strengthen when I frankly tell them that they ought to quit their jobs on ethical and/or mission grounds. (Prior to this whole AI labs business.)
Sorry, I should have expanded a bit more. The quoted part is only the first five minutes of a two-hour conversation; I definitely believe in having frank conversations about x-risk with people who are on the ground.
What I don’t believe in is jumping directly to [here’s what you ought to do], which feels deeply condescending, as opposed to staying at [let’s find our cruxes, why do you believe what you believe?].
It is also not obvious to me that having them leave is the right decision, as it might have been ~5 years ago. There seems to be delicate internal opinion politics right now in the labs where the x-risk-pilled are a non-negligible minority and their presence could significantly improve our chances of (for example) future pauses/bans, or deployment of alignment ideas, or alignment problems being solved by powerful internal models.