I agree with you that a typical instance of working at an AI lab has worse consequences in expectation than working at a tobacco company, and I think that for a person who shares all your epistemic beliefs to work in a typical role at an AI lab would indeed be a worse failure of moral character than to work at a tobacco company.
I also agree that in many cases people at AI labs have been exposed at least once to arguments which, if they had better epistemics and dedicated more time to thinking about the consequences of their work, could have convinced them that it was bad for the world for them to do such work. And I do think the failure to engage with such arguments and seriously consider them, in situations like these, is a stain on someone’s character! But I think it’s the sort of ethical failure which a majority of humans will make by default, rather than something indicative of remarkably bad morality.
Tobacco kills about 8 million people a year globally. ASI could kill about 8 billion.
I just don’t think this sort of utilitarian calculus makes sense to apply when considering the actions of people who don’t share the object-level beliefs at hand! I think people who worked to promulgate communism in the late 19th century were not unusually evil, for instance.
And I do think the failure to engage with such arguments and seriously consider them, in situations like these, is a stain on someone’s character! But I think it’s the sort of ethical failure which a majority of humans will make by default, rather than something indicative of remarkably bad morality
This also seems locally invalid. Most people in fact don’t make this ethical failure because they don’t work at AI labs, nor do they dedicate their lives to work which has nearly as much power or influence on others as this.
It does seem consistent (and agree with commonsense morality) to say that if you are smart enough to locate the levers of power in the world, and you pursue them, then you have a moral responsibility to make sure you use them right if you do get your hands on them, otherwise we will call you evil and grossly irresponsible.
I agree with you that a typical instance of working at an AI lab has worse consequences in expectation than working at a tobacco company, and I think that for a person who shares all your epistemic beliefs to work in a typical role at an AI lab would indeed be a worse failure of moral character than to work at a tobacco company.
I also agree that in many cases people at AI labs have been exposed at least once to arguments which, if they had better epistemics and dedicated more time to thinking about the consequences of their work, could have convinced them that it was bad for the world for them to do such work. And I do think the failure to engage with such arguments and seriously consider them, in situations like these, is a stain on someone’s character! But I think it’s the sort of ethical failure which a majority of humans will make by default, rather than something indicative of remarkably bad morality.
I just don’t think this sort of utilitarian calculus makes sense to apply when considering the actions of people who don’t share the object-level beliefs at hand! I think people who worked to promulgate communism in the late 19th century were not unusually evil, for instance.
This also seems locally invalid. Most people in fact don’t make this ethical failure because they don’t work at AI labs, nor do they dedicate their lives to work which has nearly as much power or influence on others as this.
It does seem consistent (and agree with commonsense morality) to say that if you are smart enough to locate the levers of power in the world, and you pursue them, then you have a moral responsibility to make sure you use them right if you do get your hands on them, otherwise we will call you evil and grossly irresponsible.