I previously agree-voted on the top-level comment and disagree-voted on your reply, but this was pretty convincing to me. I have now removed those votes.
My thinking was that some of your takeover plan seems pretty non-obvious and novel, especially the idea of creating mirror mold, so it could conceivably be useful to an AI. It’s possible that future AI will be good at executing clearly-defined tasks but bad at coming up with good ideas for achieving long-term goals—this is sort of what we have now. (I asked ChatGPT to come up with novel takeover plans, and while I can’t say its ideas definitely wouldn’t work, they seem a bit sketchy and are definitely less actionable.)
But yes, probably by the time the AI can come up with and accurately evaluate the ideas needed to create its own mirror mold from scratch, the idea of making mirror mold in the first place would be easy to come by.
I don’t agree that the ChatGPT list is less actionable, notice that all 6 scenarios listed are designed so that humans wouldn’t be aware of what was happening while slowly ceding control.
I previously agree-voted on the top-level comment and disagree-voted on your reply, but this was pretty convincing to me. I have now removed those votes.
My thinking was that some of your takeover plan seems pretty non-obvious and novel, especially the idea of creating mirror mold, so it could conceivably be useful to an AI. It’s possible that future AI will be good at executing clearly-defined tasks but bad at coming up with good ideas for achieving long-term goals—this is sort of what we have now. (I asked ChatGPT to come up with novel takeover plans, and while I can’t say its ideas definitely wouldn’t work, they seem a bit sketchy and are definitely less actionable.)
But yes, probably by the time the AI can come up with and accurately evaluate the ideas needed to create its own mirror mold from scratch, the idea of making mirror mold in the first place would be easy to come by.
I don’t agree that the ChatGPT list is less actionable, notice that all 6 scenarios listed are designed so that humans wouldn’t be aware of what was happening while slowly ceding control.