it’s the AI doing something relatively easy that is entirely a zero-sum action that removes control of the situation from humans.
I also don’t really understand the emphasis on zero-sum action. It seems pretty plausible that the AI will trade for compute with some other person around the world. My guess is, in-general, it will be easier to get access to the internet or have some kind of side-channel attack than to get root access to the datacenter the AI runs on. I do think most-likely the AI will then be capable of just taking resources instead of paying for them, but that feels like a fact contingent on quite a few details on how it goes, and trading for compute and other resources seems pretty plausible to me.
It seems pretty plausible that the AI will trade for compute with some other person around the world.
Whether this is what I’m trying to call a zero-sum action depends on whose resources it’s trading. If the plan is “spend a bunch of the capital that its creators have given it on compute somewhere else”, then I think this is importantly zero-sum—the resources are being taken from the creators of AI, which is why the AI was able to spend so many resources. If the plan was instead “produce some ten trillion dollar invention, then sell it, then use the proceeds to buy compute elsewhere”, this would seem less zero-sum, and I’m saying that I expect the first kind of thing to happen before the second.
My impression was that “zero-sum” was not used in quite the standard way. I think the idea is the AI will cause a big reassignment of Earth’s capabilities to its own control. And that that’s contrasted with the AI massively increasing its own capabilities and thus Earth’s overall capabilities.
I also don’t really understand the emphasis on zero-sum action. It seems pretty plausible that the AI will trade for compute with some other person around the world. My guess is, in-general, it will be easier to get access to the internet or have some kind of side-channel attack than to get root access to the datacenter the AI runs on. I do think most-likely the AI will then be capable of just taking resources instead of paying for them, but that feels like a fact contingent on quite a few details on how it goes, and trading for compute and other resources seems pretty plausible to me.
Whether this is what I’m trying to call a zero-sum action depends on whose resources it’s trading. If the plan is “spend a bunch of the capital that its creators have given it on compute somewhere else”, then I think this is importantly zero-sum—the resources are being taken from the creators of AI, which is why the AI was able to spend so many resources. If the plan was instead “produce some ten trillion dollar invention, then sell it, then use the proceeds to buy compute elsewhere”, this would seem less zero-sum, and I’m saying that I expect the first kind of thing to happen before the second.
My impression was that “zero-sum” was not used in quite the standard way. I think the idea is the AI will cause a big reassignment of Earth’s capabilities to its own control. And that that’s contrasted with the AI massively increasing its own capabilities and thus Earth’s overall capabilities.