This argument seems to prove too much. Are you saying that if society has learned how to do artificial induction at a superhuman level, then by the time we give a safe planner that induction subroutine, someone will have already given that induction routine to an unsafe planner? If so, what hope is there as prediction algorithms relentlessly improve? In my view, the whole point of AGI Safety research is to try to come up with ways to use powerful-enough-to-kill-you artificial induction in a way that it doesn’t kill you (and helps you achieve your other goals). But it seems you’re saying that there is a certain level of ingenuity where malicious agents will probably act with that level of ingenuity before benign agents do.
I’m saying that if you can’t protect yourself from an AI in your lab, under conditions that you carefully control, you probably couldn’t protect yourself from AI systems out there in the world.
The hope is that you can protect yourself from an AI in your lab.
But your original comment was referring to a situation in which we didn’t carefully control the AI in our lab. (By letting it have an arbitrarily long horizon). If we have lead time on other projects, I think it’s very plausible to have a situation where we couldn’t protect ourselves from our own AI if we weren’t carefully controlling the conditions, but we could protect ourselves from our own AI if we we were carefully controlling the situation, and then given our lead time, we’re not at a big risk from other projects yet.
I’m saying that if you can’t protect yourself from an AI in your lab, under conditions that you carefully control, you probably couldn’t protect yourself from AI systems out there in the world.
The hope is that you can protect yourself from an AI in your lab.
But your original comment was referring to a situation in which we didn’t carefully control the AI in our lab. (By letting it have an arbitrarily long horizon). If we have lead time on other projects, I think it’s very plausible to have a situation where we couldn’t protect ourselves from our own AI if we weren’t carefully controlling the conditions, but we could protect ourselves from our own AI if we we were carefully controlling the situation, and then given our lead time, we’re not at a big risk from other projects yet.