I think Sufficiently Scary Demos need to do something that
a) directly, clearly is capable of threatening specific world leaders from multiple nations at once, in a way that is viscerally salient to them specifically
b) but, you don’t end up going to jail for it (i.e. something like the difference between “a really good prank” and “actually hurting someone”)
c) ideally, relies as little as possible on general intelligence, as opposed to extremely powerful narrow stuff (with just enough general intelligent agency there to demonstrate that this is scary because it can be self-directed, as opposed to just being a weapon you want to make sure you control)
I think Sufficiently Scary Demos need to do something that
a) directly, clearly is capable of threatening specific world leaders from multiple nations at once, in a way that is viscerally salient to them specifically
b) but, you don’t end up going to jail for it (i.e. something like the difference between “a really good prank” and “actually hurting someone”)
c) ideally, relies as little as possible on general intelligence, as opposed to extremely powerful narrow stuff (with just enough general intelligent agency there to demonstrate that this is scary because it can be self-directed, as opposed to just being a weapon you want to make sure you control)