I think this is a pretty unhelpful frame. Most people working at an AI lab are somewhere between “person of unremarkable moral character who tells themselves a vague story about how they’re doing good things” and “deeply principled person trying their best to improve the world as best they can”. I think working at an AI lab requires less failure of moral character than, say, working at a tobacco company, for all that the former can have much worse effects on the world.
There are a few people I think it is fair to describe as actively morally bad, and willfully violating deontology—it seems likely to me that this is true of Sam Altman, for instance—but I think “evil” is just not a very helpful word here, will not usefully model the actions of AI lab employees, and will come across as obviously disingenuous to anyone who hears such rhetoric if they actually interact with any of the people you’re denigrating. If you had to be evil to end the world, the world would be a lot safer!
I think it’s fine and good to concentrate moral opprobrium at specific actions people take that are unprincipled or clear violations of deontology—companies going back on commitments, people taking on roles or supporting positions that violate principles they’ve previously expressed, people making cowardly statements that don’t accurately reflect their beliefs for the sake of currying favor. I think it’s also fine and good to try and convince people that what they’re doing is harmful, and that they should quit their jobs or turn whistleblower or otherwise change course. But the mere choice of job title is usually not a deontology violation for these people, because they don’t think it has the harms to the world you think it does! (I think at this point it is probably somewhat of a deontological violation to work in most roles at OpenAI or Meta AI even under typical x-risk-skeptical worldviews, but only one that indicates ethical mediocrity rather than ethical bankruptcy.)
(For context, I work on capabilities at Anthropic, because I think that reduces existential risk on net; I think there’s around a 25% chance that this is a horrible mistake and immensely harmful for the world. I think it’s probably quite bad for the world to work on capabilities at other AI labs.)
But the mere choice of job title is usually not a deontology violation for these people, because they don’t think it has the harms to the world you think it does!
I don’t think this step is locally valid? Or at least, in many situations, I don’t think ignorance of the consequences if your actions absolves you of responsibility for them.
As an example, if you work hard to help elect a politician who you believe was principled and good, and then when they get into office they’re a craven sellout who causes thousands of people to die, you bear some responsibility for it and for cleaning up your mess. As another example, if you work hard at a company and then it turns out the company is a scam and you’ve stolen money from all your customers, you bear some responsibility to clean up the mess and help the people whose lives your work ruined.
Relatedly, it is often the case that the right point to apply liability is when someone takes an action with a lot of downside, regardless of intent. Here are some legal examples a shoggoth gave me of holding people accountable even if they didn’t know the harm they were causing.
A company can be liable for harm caused by its products even if it followed all safety procedures.
Employers are held responsible for harms caused by employees acting within the scope of their job.
Sellers may be liable for false statements that cause harm, even if they made them in good faith.
These examples are a bit different. Anyhow, I think that if you work at a company that builds a doomsday machine, you bear some responsibility for that even if you didn’t know.
Yeah, sorry—I agree that was a bit sloppy of me. I think it is very reasonable to accuse people working at major AI labs of something like negligence / willful ignorance, and I agree that can be a pretty serious moral failing (indeed I think it’s plausibly the primary moral failing of many AI lab employees). My objection is more to the way the parent comment is connoting “evil” just from one’s employer leading to bad outcomes as if those outcomes are the known intent of such employees.
Drake—this seems like special pleading from an AI industry insider.
You wrote ‘I think working at an AI lab requires less failure of moral character than, say, working at a tobacco company, for all that the former can have much worse effects on the world.’
That doesn’t make sense to me. Tobacco kills about 8 million people a year globally. ASI could kill about 8 billion. The main reason that AI lab workers think that their moral character is better than that of tobacco industry workers is that the tobacco industry has already been morally stigmatized over the last several decades—whereas the AI industry has not yet been morally stigmatized in proportion to its likely harms.
Of course, ordinary workers in any harm-imposing industry can always make the argument that they’re good (or at least ethically mediocre) people, that they’re just following orders, trying to feed their families, weren’t aware of the harms, etc.
But that argument does not apply to smart people working in the AI industry—who have mostly already been exposed to the many arguments that AGI/ASI is a uniquely dangerous technology. And their own CEOs have already acknowledged these risks. And yet people continue to work in this industry.
Maybe a few workers at a few AI companies might be having a net positive impact in reducing AI X-risk. Maybe you’re one of the lucky few. Maybe.
The future is much much bigger than 8 billion people. Causing the extinction of humanity is much worse than killing 8 billion people. This really matters a lot for arriving at the right moral conclusions here.
Could you say a bit more about why you view the “extinction >>> 8B” as so important?
I’d have assumed that at your P(extinction), even treating extinction as just 8B deaths still vastly outweighs the possible lives saved from AI medical progress?
I don’t think it’s remotely as obvious then! If you don’t care about future people, then your key priority is to achieve immortality for the current generation, for which I do think building AGI is probably your best bet.
If it were to take 50+ years to build AGI, that would imply most people on earth have died of aging, and so you should have probably just rushed towards AGI if you think that would have been less than 50% likely to cause extinction.
People who hold this position are arguing for things like “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%”, which is a policy that if acted on consistently would more likely than not cause humanity’s extinction within 100 years (as you would be accepting a minimum of a 1% chance of death each year in exchange for faster AI development).
Share of people alive today who are expected to die*
Rough number of deaths (out of 8.23 billion)
10 years (2035)
≈ 8 %
~0.65 billion
20 years (2045)
≈ 25 %
~2.0 billion
30 years (2055)
≈ 49 %
~4.0 billion
40 years (2065)
≈ 74 %
~6.1 billion
50 years (2075)
≈ 86 %
~7.1 billion
By the logic of the future not being bigger than 8 billion people, you should lock in a policy that has a 50% chance of causing human extinction, if it allows current people alive to extend their lifespan by more than ~35 years. I am more doomy than that about AI, in that I assign much more than 50% probability that deploying superintelligence would kill everyone, but it’s definitely a claim that requires a lot more thinking through than the usual “the risk is at least 10% or so”.
Thanks for explaining that, really appreciate it! One thing I notice I’d been assuming: that “8B-only” people would have a policy like “care about the 8B people who are living today, but also the people in say 20 years who’ve been born in the intervening time period.” But that’s basically just a policy of caring about future people! Because there’s not really a difference between “future people at the point that they’ve actually been born” and “future people generally”
I have different intuitions about “causing someone not to be born” versus “waiting for someone to be born, and then killing them”. So I do think that if someone sets in motion today events that reliably end in the human race dying out in 2035, the moral cost of this might be any of
“the people alive in both 2025 and 2035”
“everyone alive in 2035”
“everyone alive in 2035, plus (perhaps with some discounting) all the kids they would have had, and the kids they would have had...”
according to different sets of intuitions. And actually I guess (1) would be rarest, so even though both (2) and (3) involve “caring about future people” in some sense, I do think they’re important to distinguish. (Caring about “future-present” versus “future-future” people?)
People who hold this position are arguing for things like “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%”, which is a policy that if acted on consistently would more likely than not cause humanity’s extinction within 100 years (as you would be accepting a minimum of a 1% chance of death each year in exchange for faster AI development).
If your goal is to maximize the expected fraction of currently alive humans who live for over 1000 years, you shouldn’t in fact make ongoingly make gambles that make it more likely than not everyone dies unless it turns out that it’s really hard to achieve this without immense risk. Perhaps that is your view: the only (realistic) way to get risk below ~50% is to delay for over 30 years. But this by no means a consensus perspective among those who are very worried about AI risk.
Separately, I don’t expect that we have many tradeoffs between elimination of human control over the future and the probability of currently alive people living for much longer other than AI, so after we eat that, there aren’t further tradeoffs to make. I think you agree with this, but your wording makes it seem as though you think there are ongoing hard tradeoffs that can’t be avoided.
I think that “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%” is not a sufficient crux for the (expensive) actions which I most want at current margins, at least if you have my empirical views. I think it is very unlikely (~7%?) that in practice we reach near the level of response (in terms of spending/delaying for misalignment risk reduction) that would be rational given this “1% / year” view and my empirical views, so my empirical views suffice to imply very different actions.
For instance, delaying for ~10 years prior to building wildly superhuman AI (while using controlled AIs at or somewhat below the level of top human experts) seems like it probably makes sense on my views but this moral perspective, especially if you can use the controlled AIs to substantially reduce/delay ongoing deaths which seems plausible. Things like massively investing in safety/alignment work also easily makes sense. There are policies that substantially reduce the risk which merely require massive effort (and which don’t particularly delay powerful AI) that we could be applying.
I do think that this policy wouldn’t be on board with the sort of long pause that (e.g.) MIRI often discusses and it does materially alter what look like the best policies (though ultimately I don’t expect to get close to these best policies anyway).
habryka—‘If you don’t care about future people’—but why would any sane person not care at all about future people?
You offer a bunch of speculative math about longevity vs extinction risk.
OK, why not run some actual analysis on which is more likely to promote longevity research: direct biomedical research on longevity, or indirect AI research on AGI in hopes that it somehow, speculatively, solves longevity?
The AI industry is currently spending something on the order of $200 billion a year on research. The biomedical research on longevity, by contrast, is currently far less than $10 billion a year.
If we spent the $200 billion a year on longevity, instead of on AI, do you seriously think that we’d do worse on solving longevity? That’s what I would advocate. And it would involve virtually no extinction risk.
You are reading things into my comments I didn’t say. I of course don’t agree, or consider it reasonable, to “not care about future people”, that’s the whole context of this subthread.
My guess is if one did adopt a position that no future people matter (which again I do not think is a reasonable position), then I think the case for slowing down AI looks a lot worse. Not bad enough to make it an obvious slam that it’s bad, and my guess overall even under that worldview it would be dumb to rush towards developing AGI like we are currently doing, but it makes the case a lot weaker. There is much less to lose if you do not care about the future.
If we spent the $200 billion a year on longevity, instead of on AI, do you seriously think that we’d do worse on solving longevity? That’s what I would advocate. And it would involve virtually no extinction risk.
My guess is for the purpose of just solving longevity, AGI investment would indeed strongly outperform general biomedical investment. Humanity just isn’t very good at turning money into medical progress on demand like this.
It seems virtuous and good to be clear about which assumptions are load-bearing to my recommended actions. If I didn’t care about the future, I would definitely be advocating for a different mix of policies, though it likely would still involve marginal AI slowdown, but my guess is less forcefully, and a bunch of slowdown-related actions would become net bad.
I agree with you that a typical instance of working at an AI lab has worse consequences in expectation than working at a tobacco company, and I think that for a person who shares all your epistemic beliefs to work in a typical role at an AI lab would indeed be a worse failure of moral character than to work at a tobacco company.
I also agree that in many cases people at AI labs have been exposed at least once to arguments which, if they had better epistemics and dedicated more time to thinking about the consequences of their work, could have convinced them that it was bad for the world for them to do such work. And I do think the failure to engage with such arguments and seriously consider them, in situations like these, is a stain on someone’s character! But I think it’s the sort of ethical failure which a majority of humans will make by default, rather than something indicative of remarkably bad morality.
Tobacco kills about 8 million people a year globally. ASI could kill about 8 billion.
I just don’t think this sort of utilitarian calculus makes sense to apply when considering the actions of people who don’t share the object-level beliefs at hand! I think people who worked to promulgate communism in the late 19th century were not unusually evil, for instance.
And I do think the failure to engage with such arguments and seriously consider them, in situations like these, is a stain on someone’s character! But I think it’s the sort of ethical failure which a majority of humans will make by default, rather than something indicative of remarkably bad morality
This also seems locally invalid. Most people in fact don’t make this ethical failure because they don’t work at AI labs, nor do they dedicate their lives to work which has nearly as much power or influence on others as this.
It does seem consistent (and agree with commonsense morality) to say that if you are smart enough to locate the levers of power in the world, and you pursue them, then you have a moral responsibility to make sure you use them right if you do get your hands on them, otherwise we will call you evil and grossly irresponsible.
I think this is a pretty unhelpful frame. Most people working at an AI lab are somewhere between “person of unremarkable moral character who tells themselves a vague story about how they’re doing good things” and “deeply principled person trying their best to improve the world as best they can”. I think working at an AI lab requires less failure of moral character than, say, working at a tobacco company, for all that the former can have much worse effects on the world.
There are a few people I think it is fair to describe as actively morally bad, and willfully violating deontology—it seems likely to me that this is true of Sam Altman, for instance—but I think “evil” is just not a very helpful word here, will not usefully model the actions of AI lab employees, and will come across as obviously disingenuous to anyone who hears such rhetoric if they actually interact with any of the people you’re denigrating. If you had to be evil to end the world, the world would be a lot safer!
I think it’s fine and good to concentrate moral opprobrium at specific actions people take that are unprincipled or clear violations of deontology—companies going back on commitments, people taking on roles or supporting positions that violate principles they’ve previously expressed, people making cowardly statements that don’t accurately reflect their beliefs for the sake of currying favor. I think it’s also fine and good to try and convince people that what they’re doing is harmful, and that they should quit their jobs or turn whistleblower or otherwise change course. But the mere choice of job title is usually not a deontology violation for these people, because they don’t think it has the harms to the world you think it does! (I think at this point it is probably somewhat of a deontological violation to work in most roles at OpenAI or Meta AI even under typical x-risk-skeptical worldviews, but only one that indicates ethical mediocrity rather than ethical bankruptcy.)
(For context, I work on capabilities at Anthropic, because I think that reduces existential risk on net; I think there’s around a 25% chance that this is a horrible mistake and immensely harmful for the world. I think it’s probably quite bad for the world to work on capabilities at other AI labs.)
I don’t think this step is locally valid? Or at least, in many situations, I don’t think ignorance of the consequences if your actions absolves you of responsibility for them.
As an example, if you work hard to help elect a politician who you believe was principled and good, and then when they get into office they’re a craven sellout who causes thousands of people to die, you bear some responsibility for it and for cleaning up your mess. As another example, if you work hard at a company and then it turns out the company is a scam and you’ve stolen money from all your customers, you bear some responsibility to clean up the mess and help the people whose lives your work ruined.
Relatedly, it is often the case that the right point to apply liability is when someone takes an action with a lot of downside, regardless of intent. Here are some legal examples a shoggoth gave me of holding people accountable even if they didn’t know the harm they were causing.
A company can be liable for harm caused by its products even if it followed all safety procedures.
Employers are held responsible for harms caused by employees acting within the scope of their job.
Sellers may be liable for false statements that cause harm, even if they made them in good faith.
These examples are a bit different. Anyhow, I think that if you work at a company that builds a doomsday machine, you bear some responsibility for that even if you didn’t know.
Yeah, sorry—I agree that was a bit sloppy of me. I think it is very reasonable to accuse people working at major AI labs of something like negligence / willful ignorance, and I agree that can be a pretty serious moral failing (indeed I think it’s plausibly the primary moral failing of many AI lab employees). My objection is more to the way the parent comment is connoting “evil” just from one’s employer leading to bad outcomes as if those outcomes are the known intent of such employees.
Drake—this seems like special pleading from an AI industry insider.
You wrote ‘I think working at an AI lab requires less failure of moral character than, say, working at a tobacco company, for all that the former can have much worse effects on the world.’
That doesn’t make sense to me. Tobacco kills about 8 million people a year globally. ASI could kill about 8 billion. The main reason that AI lab workers think that their moral character is better than that of tobacco industry workers is that the tobacco industry has already been morally stigmatized over the last several decades—whereas the AI industry has not yet been morally stigmatized in proportion to its likely harms.
Of course, ordinary workers in any harm-imposing industry can always make the argument that they’re good (or at least ethically mediocre) people, that they’re just following orders, trying to feed their families, weren’t aware of the harms, etc.
But that argument does not apply to smart people working in the AI industry—who have mostly already been exposed to the many arguments that AGI/ASI is a uniquely dangerous technology. And their own CEOs have already acknowledged these risks. And yet people continue to work in this industry.
Maybe a few workers at a few AI companies might be having a net positive impact in reducing AI X-risk. Maybe you’re one of the lucky few. Maybe.
The future is much much bigger than 8 billion people. Causing the extinction of humanity is much worse than killing 8 billion people. This really matters a lot for arriving at the right moral conclusions here.
Could you say a bit more about why you view the “extinction >>> 8B” as so important?
I’d have assumed that at your P(extinction), even treating extinction as just 8B deaths still vastly outweighs the possible lives saved from AI medical progress?
I don’t think it’s remotely as obvious then! If you don’t care about future people, then your key priority is to achieve immortality for the current generation, for which I do think building AGI is probably your best bet.
If it were to take 50+ years to build AGI, that would imply most people on earth have died of aging, and so you should have probably just rushed towards AGI if you think that would have been less than 50% likely to cause extinction.
People who hold this position are arguing for things like “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%”, which is a policy that if acted on consistently would more likely than not cause humanity’s extinction within 100 years (as you would be accepting a minimum of a 1% chance of death each year in exchange for faster AI development).
Here are ChatGPTs actuarial tables about how long the current population is expected to survive:
By the logic of the future not being bigger than 8 billion people, you should lock in a policy that has a 50% chance of causing human extinction, if it allows current people alive to extend their lifespan by more than ~35 years. I am more doomy than that about AI, in that I assign much more than 50% probability that deploying superintelligence would kill everyone, but it’s definitely a claim that requires a lot more thinking through than the usual “the risk is at least 10% or so”.
Thanks for explaining that, really appreciate it! One thing I notice I’d been assuming: that “8B-only” people would have a policy like “care about the 8B people who are living today, but also the people in say 20 years who’ve been born in the intervening time period.” But that’s basically just a policy of caring about future people! Because there’s not really a difference between “future people at the point that they’ve actually been born” and “future people generally”
I have different intuitions about “causing someone not to be born” versus “waiting for someone to be born, and then killing them”. So I do think that if someone sets in motion today events that reliably end in the human race dying out in 2035, the moral cost of this might be any of
“the people alive in both 2025 and 2035”
“everyone alive in 2035”
“everyone alive in 2035, plus (perhaps with some discounting) all the kids they would have had, and the kids they would have had...”
according to different sets of intuitions. And actually I guess (1) would be rarest, so even though both (2) and (3) involve “caring about future people” in some sense, I do think they’re important to distinguish. (Caring about “future-present” versus “future-future” people?)
If your goal is to maximize the expected fraction of currently alive humans who live for over 1000 years, you shouldn’t in fact make ongoingly make gambles that make it more likely than not everyone dies unless it turns out that it’s really hard to achieve this without immense risk. Perhaps that is your view: the only (realistic) way to get risk below ~50% is to delay for over 30 years. But this by no means a consensus perspective among those who are very worried about AI risk.
Separately, I don’t expect that we have many tradeoffs between elimination of human control over the future and the probability of currently alive people living for much longer other than AI, so after we eat that, there aren’t further tradeoffs to make. I think you agree with this, but your wording makes it seem as though you think there are ongoing hard tradeoffs that can’t be avoided.
I think that “we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%” is not a sufficient crux for the (expensive) actions which I most want at current margins, at least if you have my empirical views. I think it is very unlikely (~7%?) that in practice we reach near the level of response (in terms of spending/delaying for misalignment risk reduction) that would be rational given this “1% / year” view and my empirical views, so my empirical views suffice to imply very different actions.
For instance, delaying for ~10 years prior to building wildly superhuman AI (while using controlled AIs at or somewhat below the level of top human experts) seems like it probably makes sense on my views but this moral perspective, especially if you can use the controlled AIs to substantially reduce/delay ongoing deaths which seems plausible. Things like massively investing in safety/alignment work also easily makes sense. There are policies that substantially reduce the risk which merely require massive effort (and which don’t particularly delay powerful AI) that we could be applying.
I do think that this policy wouldn’t be on board with the sort of long pause that (e.g.) MIRI often discusses and it does materially alter what look like the best policies (though ultimately I don’t expect to get close to these best policies anyway).
habryka—‘If you don’t care about future people’—but why would any sane person not care at all about future people?
You offer a bunch of speculative math about longevity vs extinction risk.
OK, why not run some actual analysis on which is more likely to promote longevity research: direct biomedical research on longevity, or indirect AI research on AGI in hopes that it somehow, speculatively, solves longevity?
The AI industry is currently spending something on the order of $200 billion a year on research. The biomedical research on longevity, by contrast, is currently far less than $10 billion a year.
If we spent the $200 billion a year on longevity, instead of on AI, do you seriously think that we’d do worse on solving longevity? That’s what I would advocate. And it would involve virtually no extinction risk.
You are reading things into my comments I didn’t say. I of course don’t agree, or consider it reasonable, to “not care about future people”, that’s the whole context of this subthread.
My guess is if one did adopt a position that no future people matter (which again I do not think is a reasonable position), then I think the case for slowing down AI looks a lot worse. Not bad enough to make it an obvious slam that it’s bad, and my guess overall even under that worldview it would be dumb to rush towards developing AGI like we are currently doing, but it makes the case a lot weaker. There is much less to lose if you do not care about the future.
My guess is for the purpose of just solving longevity, AGI investment would indeed strongly outperform general biomedical investment. Humanity just isn’t very good at turning money into medical progress on demand like this.
It seems virtuous and good to be clear about which assumptions are load-bearing to my recommended actions. If I didn’t care about the future, I would definitely be advocating for a different mix of policies, though it likely would still involve marginal AI slowdown, but my guess is less forcefully, and a bunch of slowdown-related actions would become net bad.
I agree with you that a typical instance of working at an AI lab has worse consequences in expectation than working at a tobacco company, and I think that for a person who shares all your epistemic beliefs to work in a typical role at an AI lab would indeed be a worse failure of moral character than to work at a tobacco company.
I also agree that in many cases people at AI labs have been exposed at least once to arguments which, if they had better epistemics and dedicated more time to thinking about the consequences of their work, could have convinced them that it was bad for the world for them to do such work. And I do think the failure to engage with such arguments and seriously consider them, in situations like these, is a stain on someone’s character! But I think it’s the sort of ethical failure which a majority of humans will make by default, rather than something indicative of remarkably bad morality.
I just don’t think this sort of utilitarian calculus makes sense to apply when considering the actions of people who don’t share the object-level beliefs at hand! I think people who worked to promulgate communism in the late 19th century were not unusually evil, for instance.
This also seems locally invalid. Most people in fact don’t make this ethical failure because they don’t work at AI labs, nor do they dedicate their lives to work which has nearly as much power or influence on others as this.
It does seem consistent (and agree with commonsense morality) to say that if you are smart enough to locate the levers of power in the world, and you pursue them, then you have a moral responsibility to make sure you use them right if you do get your hands on them, otherwise we will call you evil and grossly irresponsible.