But “being able to threaten the whole world” is a moving target, because technology advances for the world too.
This is a bet that every single existential threat ever made accessible to individuals and small groups through advanced technology is defense-dominant. I wouldn’t put my money on it.
The trend in history is that existential threats and WMDs used to be impossible to build, but as technology improves it becomes easier and easier, and I would expect this trend to continue. If even a single one of them turns out to be very offense-dominant then society is at mercy of any small group willing to build massively destructive WMDs in secret.
For the second article I think it might be wrong on a few key points:
There is a heavy selection pressure in which businesses succeed and end up grabbing power. Perhaps this selection is less than in politics, but nonetheless favors those willing to do “what is necessary”. The article acknowledges it, but IMO underestimates it.
It ignores parts of human nature adjacent to mating instincts that can lead men to abuse power when they have it. (It’s possible that this might not be a strong effect due to some types of powerful actors being more rational than average, but the article ignores it entirely while making other human nature arguments, so I think it would be wise to also point this out.)
It fails to consider what someone with AI needs to do with said AI in order to grab absolute power. I think that “I got to ASI first and therefore I own everything” is not true by default, and in fact taking over the world with ASI might require one to do a lot of short-term evil things (at the very least including undermining every single democracy on Earth). It’s possible that someone might get ASI and just decide not to take over the world, losing their window of opportunity in the process as others catch up, similar to how the United States got nukes first but didn’t decide to try to conquer everyone in say 1948 by threatening to nuke them. If good actors with ASI are less likely to take over than bad actors (possibly because of deontological moral red lines), then that means less probability of a good world conditional on a few people taking over the world with ASI.
On your first, I agree and I think your phrasing is better than mine was in section 5.1 of the full post, although I also focused on the dangers of RSI/creating ASI from AGI. Framing it as needing defense to be dominant in all domains (without universal surveillance, which is sort of more like offense) is a very good way to think about the problem with d/acc as a long term strategy. It doesn’t help that most areas actually seem offense-dominant; but even if I’m mistaken about that, you’d need ALL domains of confict to be defense-dominant.
On the others:
1) I don’t think business dynamics are relevant? But yes I do agree that viciousness does often help gain power. But I must note that it also often doesn’t. Power is often accumulated by those who are perceived as trustworthy and cooperative; that’s how they get broad support from many people to gain power. And corporate structures involve a lot more personal knowledge than the democratic let alone undemocratic government structures.
2) I don’t think mating instincts are a distinct case from the analysis I make. If the emperor isn’t highly sadistic, he’d rather have “mates” who like and respect him. So it follows the same empathy-minus-sadism balance as other areas of life. There are a lot of drives and I didn’t try to cover them individually, even in the long version.
3) I agree that good people might fail to take over even when they could do it bloodlessly. But once they have an AGI to tell them “so yeah if you don’t take over this is probably gonna go much worse and maybe everyone dies or gets a permanent dystopia instead of a glorious future” I think they’ll get over their deontological hangups and do the sensible thing and save the future. At least to the extent their AGI is smart enough to be reliable on that judgment call.
I basically disagree with the first article.
This is a bet that every single existential threat ever made accessible to individuals and small groups through advanced technology is defense-dominant. I wouldn’t put my money on it.
The trend in history is that existential threats and WMDs used to be impossible to build, but as technology improves it becomes easier and easier, and I would expect this trend to continue. If even a single one of them turns out to be very offense-dominant then society is at mercy of any small group willing to build massively destructive WMDs in secret.
For the second article I think it might be wrong on a few key points:
There is a heavy selection pressure in which businesses succeed and end up grabbing power. Perhaps this selection is less than in politics, but nonetheless favors those willing to do “what is necessary”. The article acknowledges it, but IMO underestimates it.
It ignores parts of human nature adjacent to mating instincts that can lead men to abuse power when they have it. (It’s possible that this might not be a strong effect due to some types of powerful actors being more rational than average, but the article ignores it entirely while making other human nature arguments, so I think it would be wise to also point this out.)
It fails to consider what someone with AI needs to do with said AI in order to grab absolute power. I think that “I got to ASI first and therefore I own everything” is not true by default, and in fact taking over the world with ASI might require one to do a lot of short-term evil things (at the very least including undermining every single democracy on Earth). It’s possible that someone might get ASI and just decide not to take over the world, losing their window of opportunity in the process as others catch up, similar to how the United States got nukes first but didn’t decide to try to conquer everyone in say 1948 by threatening to nuke them. If good actors with ASI are less likely to take over than bad actors (possibly because of deontological moral red lines), then that means less probability of a good world conditional on a few people taking over the world with ASI.
Thanks for engaging!
On your first, I agree and I think your phrasing is better than mine was in section 5.1 of the full post, although I also focused on the dangers of RSI/creating ASI from AGI. Framing it as needing defense to be dominant in all domains (without universal surveillance, which is sort of more like offense) is a very good way to think about the problem with d/acc as a long term strategy. It doesn’t help that most areas actually seem offense-dominant; but even if I’m mistaken about that, you’d need ALL domains of confict to be defense-dominant.
On the others:
1) I don’t think business dynamics are relevant? But yes I do agree that viciousness does often help gain power. But I must note that it also often doesn’t. Power is often accumulated by those who are perceived as trustworthy and cooperative; that’s how they get broad support from many people to gain power. And corporate structures involve a lot more personal knowledge than the democratic let alone undemocratic government structures.
2) I don’t think mating instincts are a distinct case from the analysis I make. If the emperor isn’t highly sadistic, he’d rather have “mates” who like and respect him. So it follows the same empathy-minus-sadism balance as other areas of life. There are a lot of drives and I didn’t try to cover them individually, even in the long version.
3) I agree that good people might fail to take over even when they could do it bloodlessly. But once they have an AGI to tell them “so yeah if you don’t take over this is probably gonna go much worse and maybe everyone dies or gets a permanent dystopia instead of a glorious future” I think they’ll get over their deontological hangups and do the sensible thing and save the future. At least to the extent their AGI is smart enough to be reliable on that judgment call.