I agree, I think there is a common part of the story that goes “once connected to the internet, the AI rapidly takes over a large number of computers, significantly amplifying its power”. My credence that this could happen has gone way up over the last 10 years or so. Also my credence that an entity could infiltrate a very large number of machines without anyone noticing has also gone up.
In conclusion, we argue that a compact worm that begins with a list including all likely vulnerable addresses, and that has initial knowledge of some vulnerable sites with high-bandwidth links, appears able to infect almost all vulnerable servers on the Internet in less than thirty seconds.
...we argue that a compact worm...appears able to infect almost all vulnerable servers on the Internet in less than thirty seconds.
What’s important with respect to taking over the world is the amount and nature of control that can be gained by any given exploit.
Stuxnet was allegedly able to ruin one-fifth of Iran’s nuclear centrifuges. Causing such damage is far from taking useful control of important infrastructure. It is not possible to e.g. remote control the Iranian nuclear program in order to build nukes and rockets, which are then remotely launched to take out the Iranian defense capabilities.
I agree, I think there is a common part of the story that goes “once connected to the internet, the AI rapidly takes over a large number of computers, significantly amplifying its power”. My credence that this could happen has gone way up over the last 10 years or so. Also my credence that an entity could infiltrate a very large number of machines without anyone noticing has also gone up.
Whenever you see the words “Internet of things”, think “unfixable Heartbleed everywhere forever”.
Hasn’t something much like this already happened?
Staniford, Paxson & Weaver 2002
What’s important with respect to taking over the world is the amount and nature of control that can be gained by any given exploit.
Stuxnet was allegedly able to ruin one-fifth of Iran’s nuclear centrifuges. Causing such damage is far from taking useful control of important infrastructure. It is not possible to e.g. remote control the Iranian nuclear program in order to build nukes and rockets, which are then remotely launched to take out the Iranian defense capabilities.