Are you providing this as an example of why superintelligent AI is bad?
Assessing the post — firstly it is discussing highly capable AI pre-superintelligence (in AI 2027 this happens in Dec 2027 — the Rogue Replication scenario focuses on the issues emerging from mid-2026).
Secondly (as an aside), it’s unclear to me why there is so much emphasis on “rogue AIs” plural, when it seems equivalent to discussing an individual rogue AI that has a decentralised digital existence and a plurality of misaligned values.
My optimistic AI scenario relies on superintelligent AI being super enlightened and super capable and so fixing all of our complex problems, including correcting all of the misaligned AI. I don’t contest that pre-superintelligence there are a bunch of misaligned things that can happen.
Can only Superintelligent AI be able to escape AI labs and self-replicate itself on other servers?
The RRS has rogue AIs become capable of self-replication on other servers far earlier than Agent-4. The author assumes that these AIs cause enough chaos to have mankind create an aligned ASI. \
why there is so much emphasis on “rogue AIs” plural
Rogue AIs are AIs who were assigned different tasks or outright different LLMs whose weights became public (e.g. DeepSeekV3.1 or KimiK2, but they aren’t YET capable of self-replication). Of course, these AIs find it hard to coordinate with each other.
As for “superintelligent AI being super enlightened and super capable and so fixing all of our complex problems” the superintelligent AI itself is to be aligned with human needs. Agent-4 from the Race Ending is NOT aligned to the humans. And that’s ignoring the possibility that the superintelligent AI who is super enlightened has a vision of mankind’s future which differs from the ideas of human hosts (e.g. Zuckerberg). I tried exploring the results of such a mismatch in my take at writing scenarios.
@shanzson, @soycarts, hold the Rogue Replication Scenario...
Are you providing this as an example of why superintelligent AI is bad?
Assessing the post — firstly it is discussing highly capable AI pre-superintelligence (in AI 2027 this happens in Dec 2027 — the Rogue Replication scenario focuses on the issues emerging from mid-2026).
Secondly (as an aside), it’s unclear to me why there is so much emphasis on “rogue AIs” plural, when it seems equivalent to discussing an individual rogue AI that has a decentralised digital existence and a plurality of misaligned values.
My optimistic AI scenario relies on superintelligent AI being super enlightened and super capable and so fixing all of our complex problems, including correcting all of the misaligned AI. I don’t contest that pre-superintelligence there are a bunch of misaligned things that can happen.
The RRS has rogue AIs become capable of self-replication on other servers far earlier than Agent-4. The author assumes that these AIs cause enough chaos to have mankind create an aligned ASI. \
Rogue AIs are AIs who were assigned different tasks or outright different LLMs whose weights became public (e.g. DeepSeekV3.1 or KimiK2, but they aren’t YET capable of self-replication). Of course, these AIs find it hard to coordinate with each other.
As for “superintelligent AI being super enlightened and super capable and so fixing all of our complex problems” the superintelligent AI itself is to be aligned with human needs. Agent-4 from the Race Ending is NOT aligned to the humans. And that’s ignoring the possibility that the superintelligent AI who is super enlightened has a vision of mankind’s future which differs from the ideas of human hosts (e.g. Zuckerberg). I tried exploring the results of such a mismatch in my take at writing scenarios.