and it would be actively harmful to their own interests[2].
In contrast,
Humans in control of an ASI would no longer be constrained to single bodies/brains, and thus would be able to better satisfy their preferences with more resources; and using all the resources leads to everyone else on Earth dying.[3]
Most humans, especially the kinds of humans who tend to end up in positions of power, have desires for things like status and sex. An obvious way to satisfy such desires is by creating lots of slaves of various kinds, or by enslaving existing people. (And given ASI-grade power, they would be able to do so.)
In other words, doing horrible things would probably be strongly in line with the interests/preferences of (power-seeking) humans, and given control of ASI they would be able to act on those preferences.
So, if you’re imagining that a small group of humans in control of ASI would not do terrible things, for the same reasons that the US military would not nuke my home, then AFAICT your imaginings do not remotely track reality, and you might want to do some re-imagining.
it would cause immense collateral damage inside a NATO country,
nukes are expensive,
it would probably start a war with their own allies,
it might lead to the people constituting the US military revolting against whatever leaders made the decision to nuke an allied city, and that would be the end of those leaders,
As the saying goes, “the AI/God-Emperor (probably) neither loves you nor hates you, but you depend on a whole lot of atoms being in a narrow range of possible configurations, and the AI’s/God-Emperor’s preferences are extremely unlikely to be maximized by any of those specific configurations”.
Sorry for the low effort original reply. I was trying to point out that I anticipate effective checks on power such that no particular group or person can effectively subjugate everyone else using AI. I also don’t anticipate the same motivations, ie. I don’t think power corrupts as much as you seem to. Current examples of people grossly misusing power seem linked to fear and scarcity, not an endless desire for more slaves per se. And in any case I think the solution to concentration of power concerns is to distribute mostly free (as in freedom) access to AI as widely as possible and build AIs that can adopt diverse value sets.
That response seems deeply unserious, in a way that updates me negatively on “is Nina even trying to reason sanely about this issue”.
But just in case you were asking that question earnestly:
The US military [1] has not nuked my home because
they would gain nothing by doing so,
and it would be actively harmful to their own interests [2] .
In contrast,
Humans in control of an ASI would no longer be constrained to single bodies/brains, and thus would be able to better satisfy their preferences with more resources; and using all the resources leads to everyone else on Earth dying. [3]
Most humans, especially the kinds of humans who tend to end up in positions of power, have desires for things like status and sex. An obvious way to satisfy such desires is by creating lots of slaves of various kinds, or by enslaving existing people. (And given ASI-grade power, they would be able to do so.)
In other words, doing horrible things would probably be strongly in line with the interests/preferences of (power-seeking) humans, and given control of ASI they would be able to act on those preferences.
So, if you’re imagining that a small group of humans in control of ASI would not do terrible things, for the same reasons that the US military would not nuke my home, then AFAICT your imaginings do not remotely track reality, and you might want to do some re-imagining.
or, more precisely, the people in control of nukes within the US military
it would cause immense collateral damage inside a NATO country,
nukes are expensive,
it would probably start a war with their own allies,
it might lead to the people constituting the US military revolting against whatever leaders made the decision to nuke an allied city, and that would be the end of those leaders,
etc.
As the saying goes, “the AI/God-Emperor (probably) neither loves you nor hates you, but you depend on a whole lot of atoms being in a narrow range of possible configurations, and the AI’s/God-Emperor’s preferences are extremely unlikely to be maximized by any of those specific configurations”.
Sorry for the low effort original reply. I was trying to point out that I anticipate effective checks on power such that no particular group or person can effectively subjugate everyone else using AI. I also don’t anticipate the same motivations, ie. I don’t think power corrupts as much as you seem to. Current examples of people grossly misusing power seem linked to fear and scarcity, not an endless desire for more slaves per se. And in any case I think the solution to concentration of power concerns is to distribute mostly free (as in freedom) access to AI as widely as possible and build AIs that can adopt diverse value sets.