I can kinda scan through the logic step by step, e.g. “we need somewhat capable systems in order to study alignment”, but something about it doesn’t make sense. Like, aren’t we worried about AGI?
Most people just don’t have the doom-by-default outlook that’s prevalent in the LW/MIRI sphere. Sure, they are worried about AGI, but they also want cancer cured, poverty eradicated, catgirl volcano lairs, etc. Building AGI seems to be the only way we’re getting that stuff any time soon, and in general one of the few remaining avenues for techno-optimism amid the bleak degrowth-and-despair cultural landscape, in the face of which accepting some substantial-but-not-overwhelming level of risk is a no-brainer.
Even they likely think that the counterfactual impact of any (ex ante reasonable) decision in their power could contribute only a minor fraction of that number. Of course, a case could be made that this is pernicious motivated reasoning, I’m only saying that it’s consistent with my model of human thinking and behavior.
Most people just don’t have the doom-by-default outlook that’s prevalent in the LW/MIRI sphere. Sure, they are worried about AGI, but they also want cancer cured, poverty eradicated,
catgirl volcano lairs, etc. Building AGI seems to be the only way we’re getting that stuff any time soon, and in general one of the few remaining avenues for techno-optimism amid the bleak degrowth-and-despair cultural landscape, in the face of which accepting some substantial-but-not-overwhelming level of risk is a no-brainer.The part that doesn’t go through is when the people are people discussed the OP, who do thing there’s some huge (>10%) chance of extinction from AGI.
Even they likely think that the counterfactual impact of any (ex ante reasonable) decision in their power could contribute only a minor fraction of that number. Of course, a case could be made that this is pernicious motivated reasoning, I’m only saying that it’s consistent with my model of human thinking and behavior.