Instead of the Singularity Institute having to prove that AGI can potentially be dangerous, really AGI researchers should have to prove the opposite.
How’s about we prove that teens texting can not result in emergence of hivemind that would subsequently invent better hardware to run itself on, and rid of everyone?
How’s about you take AIXI, and analyze it, and see that it doesn’t relate itself to it’s computational substrate, subsequently being unable to understand self preservation? There are other, much more relevant ways of being safe than “ohh it talks so moral”.
How’s about we prove that teens texting can not result in emergence of hivemind that would subsequently invent better hardware to run itself on, and rid of everyone?
How’s about you take AIXI, and analyze it, and see that it doesn’t relate itself to it’s computational substrate, subsequently being unable to understand self preservation? There are other, much more relevant ways of being safe than “ohh it talks so moral”.