If a rogue AI is discovered early, we could end up in a war where the AGI has a huge intelligence advantage, but humans have a huge resource advantage.
In that scenario, it seems to me that enough abstractions break down that the analogy to the Stockfish experiment no longer works. Like talking about a conflict of AGI vs. “humans” as two agents in a 2-player game, rather than AGI vs. a collection of exploitable agents.
But I want to focus on the “resource” abstraction here. First of all, “ownership” of resources seems irrelevant; that’s mostly a legal concept, and seems like an irrelevant fiction in this scenario. What matters more in a conflict is possession of resources, i.e. who can actually control and command resources.
And here things get tricky. Maybe humans ostensibly possess nukes, but it’s not clear whether they would in practice be able to employ them against the AI (what would you even target?), or rather see them employed against them via hacks or social engineering. Humans certainly possess a lot of computers, but can quickly lose possession over a significant chunk of them when they’re turned into botnets. And so on.
Overall, in terms of security mindset, it does not make too much sense to me to think in terms of a conflict of Team AGI vs Team Humanity, where one side has an enormous resource advantage.
In that scenario, it seems to me that enough abstractions break down that the analogy to the Stockfish experiment no longer works. Like talking about a conflict of AGI vs. “humans” as two agents in a 2-player game, rather than AGI vs. a collection of exploitable agents.
But I want to focus on the “resource” abstraction here. First of all, “ownership” of resources seems irrelevant; that’s mostly a legal concept, and seems like an irrelevant fiction in this scenario. What matters more in a conflict is possession of resources, i.e. who can actually control and command resources.
And here things get tricky. Maybe humans ostensibly possess nukes, but it’s not clear whether they would in practice be able to employ them against the AI (what would you even target?), or rather see them employed against them via hacks or social engineering. Humans certainly possess a lot of computers, but can quickly lose possession over a significant chunk of them when they’re turned into botnets. And so on.
Overall, in terms of security mindset, it does not make too much sense to me to think in terms of a conflict of Team AGI vs Team Humanity, where one side has an enormous resource advantage.