That wouldn’t be cool, it would be very Ted Kaczynski-ish.
Is not embarrassing yourself by looking a bit like Ted Kaczynski your only terminal value?
Just because you’re making an AGI that can’t be “proven friendly,” it might be friendly.
I suggest thinking inside the box more. The hypothetical is ‘What if FAI is impossible?‘, not ‘What if we can’t prove that FAI is actual?’. But all your suggestions are attempts to resist that premise, not attempts to explore its consequences. One of the basic concerns of the EA movement is that it’s very dangerous not to be able to seriously entertain worst-case scenarios.
Also: maybe we could reason with it before it killed us all, and possibly change its mind.
If it kills us all in less than the time it takes to construct a reasoned paragraph in any human language—which is possibly the default scenario, the most common one—then that will be difficult.
More importantly, any relevant argument we could come up with will almost certainly have already been thought of by the UFAI, and presented with far more rigor than any human could. You should be more confident that a chimpanzee could beat Garry Kasparov in chess than that a human could outwit a post-FOOM AI.
Huh?
Is not embarrassing yourself by looking a bit like Ted Kaczynski your only terminal value?
I suggest thinking inside the box more. The hypothetical is ‘What if FAI is impossible?‘, not ‘What if we can’t prove that FAI is actual?’. But all your suggestions are attempts to resist that premise, not attempts to explore its consequences. One of the basic concerns of the EA movement is that it’s very dangerous not to be able to seriously entertain worst-case scenarios.
More, But there’s still a chance, right? isn’t the right way to think about any question. The question is whether the probability is low enough to be worth the risk, not whether the probability is nonzero.
If it kills us all in less than the time it takes to construct a reasoned paragraph in any human language—which is possibly the default scenario, the most common one—then that will be difficult.
More importantly, any relevant argument we could come up with will almost certainly have already been thought of by the UFAI, and presented with far more rigor than any human could. You should be more confident that a chimpanzee could beat Garry Kasparov in chess than that a human could outwit a post-FOOM AI.