My Cyberwarfare Concerns: A disorganized and incomplete list
A lot of internet infrastructure (e.g. BGP / routing) basically works because all the big players mostly cooperate. There have been minor incidents and attacks but nothing major so far. It seems likely to be the case that if a major superpower was backed into a corner, it could massively disrupt the internet, which would be bad.
Cyberwar has a lot of weird asymmetries where the largest attack surfaces are private companies (not militaries/governments). This gets weirder when private companies are multinational. (Is an attack on google an attack on ireland? USA? Neither/both?)
It’s unclear who is on whose side. The Snowden leaks showed that american intelligence was hacking american companies private fibers on american soil, and the trust still hasn’t recovered. It’s a low-trust environment out there, which seems (to me) to make conflict more likely to start, and harder to contain and extinguish once started.
There is no good international “law of war” with regards to cyberwarfare. There are some works-in-progress which have been slowly advancing, but there’s nothing like the geneva convention yet. Right now existing cyber conflicts haven’t really pushed the “what is an illegal attack” sense (in the way that land mines are “illegal” in war), and the lack of clear guidance here means that in an all-out conflict there isn’t much in the way of clear limitations.
Many cyber attacks are intentionally vague or secret in origin. Some of this is because groups are distributed and only loosely connected to national powers (e.g. via funding, blind eyes, etc) and some because it’s practically useful to have plausible deniability. This really gets in the way of any sort of “cease fire” or armistice agreements—if a country comes to the peace treaty table for a given cyber conflict, this might end up implicating them as the source of an attack.
Expanding the last point more, I’m worried that there are a lot of “ratchet-up” mechanisms for cyberwarfare, but very few “ratchet-down” mechanisms. All of these worries somewhat contribute to a situation where if the currently-low-grade-burning-cyberwar turns into more of an all-out cyberwar, we’ll have very few tools for deescalation.
Relating this to my concerns about AGI safety, I think an ‘all-out cyberwar’ (or at least a much larger scale one) is one of the primary ways to trigger an AGI weapons development program. Right now it’s not clear to me that much of weapons development budget is spent on cyberweapons (as opposed to other capabilities like SIGINT), but a large-scale cyberwar seems like a reason to invest more. The more money is spent on cyberweapons development, the more likely I think it is that an AGI weapons program is started. I’m not optimistic about the alignment or safety of an AGI weapons program.
Maybe more to come in the future but that’s it for now.
Sure, I’m not optimistic about the alignment of cyberweapons, but optimism about them not being too general seems more warranted. They would be another case of people wanting results NOW, ie hacking together existing techniques.
Some of this is because groups are distributed and only loosely connected to national powers (e.g. via funding, blind eyes, etc) and some because it’s practically useful to have plausible deniability.
Apart from groups whose purpose is attacking, the security teams at the FANG companies are likely also capable of attacking if they wanted and employ some of the most capable individuals.
We need a debate about what’s okay for a Google security person to do in their 20% time. Is it okay to join the conflict and defend Ukrainian cyber assets? Is it okay to hack Russian targets in the process? Should the FANG companies explicitly order their employees to keep out of the conflict?
My Cyberwarfare Concerns: A disorganized and incomplete list
A lot of internet infrastructure (e.g. BGP / routing) basically works because all the big players mostly cooperate. There have been minor incidents and attacks but nothing major so far. It seems likely to be the case that if a major superpower was backed into a corner, it could massively disrupt the internet, which would be bad.
Cyberwar has a lot of weird asymmetries where the largest attack surfaces are private companies (not militaries/governments). This gets weirder when private companies are multinational. (Is an attack on google an attack on ireland? USA? Neither/both?)
It’s unclear who is on whose side. The Snowden leaks showed that american intelligence was hacking american companies private fibers on american soil, and the trust still hasn’t recovered. It’s a low-trust environment out there, which seems (to me) to make conflict more likely to start, and harder to contain and extinguish once started.
There is no good international “law of war” with regards to cyberwarfare. There are some works-in-progress which have been slowly advancing, but there’s nothing like the geneva convention yet. Right now existing cyber conflicts haven’t really pushed the “what is an illegal attack” sense (in the way that land mines are “illegal” in war), and the lack of clear guidance here means that in an all-out conflict there isn’t much in the way of clear limitations.
Many cyber attacks are intentionally vague or secret in origin. Some of this is because groups are distributed and only loosely connected to national powers (e.g. via funding, blind eyes, etc) and some because it’s practically useful to have plausible deniability. This really gets in the way of any sort of “cease fire” or armistice agreements—if a country comes to the peace treaty table for a given cyber conflict, this might end up implicating them as the source of an attack.
Expanding the last point more, I’m worried that there are a lot of “ratchet-up” mechanisms for cyberwarfare, but very few “ratchet-down” mechanisms. All of these worries somewhat contribute to a situation where if the currently-low-grade-burning-cyberwar turns into more of an all-out cyberwar, we’ll have very few tools for deescalation.
Relating this to my concerns about AGI safety, I think an ‘all-out cyberwar’ (or at least a much larger scale one) is one of the primary ways to trigger an AGI weapons development program. Right now it’s not clear to me that much of weapons development budget is spent on cyberweapons (as opposed to other capabilities like SIGINT), but a large-scale cyberwar seems like a reason to invest more. The more money is spent on cyberweapons development, the more likely I think it is that an AGI weapons program is started. I’m not optimistic about the alignment or safety of an AGI weapons program.
Maybe more to come in the future but that’s it for now.
Sure, I’m not optimistic about the alignment of cyberweapons, but optimism about them not being too general seems more warranted. They would be another case of people wanting results NOW, ie hacking together existing techniques.
Apart from groups whose purpose is attacking, the security teams at the FANG companies are likely also capable of attacking if they wanted and employ some of the most capable individuals.
We need a debate about what’s okay for a Google security person to do in their 20% time. Is it okay to join the conflict and defend Ukrainian cyber assets? Is it okay to hack Russian targets in the process? Should the FANG companies explicitly order their employees to keep out of the conflict?