as discussed on the other thread, i’m imagining other countries have access to powerful AI as well. Or even equally good AI if they are sold access or if another actor catches up after the SIE fizzles (as it must)
OK. And that’s a crux then, I think that other countries mostly won’t have access to similarly-powerful AI before it’s too late. I agree they’d catch up eventually if left alone, but I don’t expect them to be left alone. If the first faction to get to superintelligence calculates that, six months from now, a rival faction will have similarly-powerful AI, and therefore be able to compete with them during the industrial explosion etc. to divide up the world between them, then they’ll think “if we can slow down that rival faction, we won’t have to share power over the world with them” and then they’ll think “I wonder if there are ways to slow down rival factions whilst we consolidate our advantage and do our industrial explosion… perhaps by using some weapons or political machinations cooked up in the next three months or so?” and they’ll probably think of something fairly effective, since they are superintelligences without any significant rivals to contend with.
Do you think the same for a company within the US? That with a 6 month, or even just a 3 month going off recent trends, lead it would find a way to sabotage other companies?
I think it’s plausible, but:
cyber attacks might be detectable and they’re unequivocally illegal
political manoeuvrings take time and there’s a pretty strong status quo bias in politics (hard to pass legislation)
and the govt typically doesn’t private companies to have monopolies on critical tech, and even with a private-public partnership my understanding is govt would want multiple companies involved
and the govt might get very involved if there’s a public-private partnership, which could threaten company leadership
(I think a remerging crux here might be the power of AI’s persuasion+strategy skills)
Yes, I do think the same for a company within the US. I think (a) it might be willing to do illegal things (companies do illegal things all the time when they think they can probably get away with it) and (b) some political maneuverings take very little time indeed; think about how much has happened in US politics since Trump took office less than a year ago. Elon’s star rose and fell for example, DOGE happened, etc. And this is in the before times; during the singularity there’ll be a general sense of crisis and emergency that makes 2025 feel like boring business as usual. A particular move I find all too plausible is “We should consolidate our compute into one big project, that shares model weights and info etc., so that we can go faster and beat china. (and, quietly, in the fine print, the AIs that should run on most of this compute should be the smartest ones and/or the people in charge should be the leadership of the most advanced company.)” In other words, we should basically grab all the compute from rival companies and give it to our project, though legally that’s not what’s happening and the narrative sugarcoats it.
Part where I am confused is why is this scenario considered as distinct over the standard ASI misalignment problem? A superintelligence that economically destroys and subjugates every country except ,perhaps, the country where it is based in is pretty close to the standard paperclip outcome right?
Whether I am turned into paperclips or completely enslaved by US-based superintelligence is rather trivial difference IMO and I think it could be treated as another variant of alignment failure.
as discussed on the other thread, i’m imagining other countries have access to powerful AI as well. Or even equally good AI if they are sold access or if another actor catches up after the SIE fizzles (as it must)
OK. And that’s a crux then, I think that other countries mostly won’t have access to similarly-powerful AI before it’s too late. I agree they’d catch up eventually if left alone, but I don’t expect them to be left alone. If the first faction to get to superintelligence calculates that, six months from now, a rival faction will have similarly-powerful AI, and therefore be able to compete with them during the industrial explosion etc. to divide up the world between them, then they’ll think “if we can slow down that rival faction, we won’t have to share power over the world with them” and then they’ll think “I wonder if there are ways to slow down rival factions whilst we consolidate our advantage and do our industrial explosion… perhaps by using some weapons or political machinations cooked up in the next three months or so?” and they’ll probably think of something fairly effective, since they are superintelligences without any significant rivals to contend with.
Do you think the same for a company within the US? That with a 6 month, or even just a 3 month going off recent trends, lead it would find a way to sabotage other companies?
I think it’s plausible, but:
cyber attacks might be detectable and they’re unequivocally illegal
political manoeuvrings take time and there’s a pretty strong status quo bias in politics (hard to pass legislation)
and the govt typically doesn’t private companies to have monopolies on critical tech, and even with a private-public partnership my understanding is govt would want multiple companies involved
and the govt might get very involved if there’s a public-private partnership, which could threaten company leadership
(I think a remerging crux here might be the power of AI’s persuasion+strategy skills)
Yes, I do think the same for a company within the US. I think (a) it might be willing to do illegal things (companies do illegal things all the time when they think they can probably get away with it) and (b) some political maneuverings take very little time indeed; think about how much has happened in US politics since Trump took office less than a year ago. Elon’s star rose and fell for example, DOGE happened, etc. And this is in the before times; during the singularity there’ll be a general sense of crisis and emergency that makes 2025 feel like boring business as usual. A particular move I find all too plausible is “We should consolidate our compute into one big project, that shares model weights and info etc., so that we can go faster and beat china. (and, quietly, in the fine print, the AIs that should run on most of this compute should be the smartest ones and/or the people in charge should be the leadership of the most advanced company.)” In other words, we should basically grab all the compute from rival companies and give it to our project, though legally that’s not what’s happening and the narrative sugarcoats it.
Part where I am confused is why is this scenario considered as distinct over the standard ASI misalignment problem? A superintelligence that economically destroys and subjugates every country except ,perhaps, the country where it is based in is pretty close to the standard paperclip outcome right?
Whether I am turned into paperclips or completely enslaved by US-based superintelligence is rather trivial difference IMO and I think it could be treated as another variant of alignment failure.