A common theme I’ve read from folks who’re less concerned about (near) future AI risks is an offense-defense balance. AGI-level offensive capabilities may be offset by AGI-level defensive capabilities. However, we know that LLMs have jagged capability levels; they are excellent at some capabilities but terrible at adjacent ones. For instance, are LLMs excellent at generating malware but bad at malware detection? Understanding settings where the nature of defenses makes them ill-suited for automation, but where offense is easily automated, seems critical for societal resilience.
For instance, are LLMs excellent at generating malware but bad at malware detection?
At current tech levels defense seems favored for anyone who is actually trying: LLMs are much better at looking in a codebase for smells of insecure code than they are at the multi-step process of developing an exploit that requires chaining together multiple gadgets correctly and frequently with no intermediate feedback. Concretely, “there is a vulnerability in this codebase: find it and patch it” seems like an easier task for LLMs than “there is a vulnerability in this codebase: find it and develop a working exploit”.
That said, the majority of exploits target systems which do not meet the “someone is actually trying to make this system secure” standard. If offense gets easier and defense remains at zero, offense becomes increasingly favored.
Regarding cybersecurity specifically, I have the unpopular opinion that it’s good actually that companies and govts are no longer able to protect their own privacy (or the privacy of their users).
A common theme I’ve read from folks who’re less concerned about (near) future AI risks is an offense-defense balance. AGI-level offensive capabilities may be offset by AGI-level defensive capabilities. However, we know that LLMs have jagged capability levels; they are excellent at some capabilities but terrible at adjacent ones. For instance, are LLMs excellent at generating malware but bad at malware detection? Understanding settings where the nature of defenses makes them ill-suited for automation, but where offense is easily automated, seems critical for societal resilience.
At current tech levels defense seems favored for anyone who is actually trying: LLMs are much better at looking in a codebase for smells of insecure code than they are at the multi-step process of developing an exploit that requires chaining together multiple gadgets correctly and frequently with no intermediate feedback. Concretely, “there is a vulnerability in this codebase: find it and patch it” seems like an easier task for LLMs than “there is a vulnerability in this codebase: find it and develop a working exploit”.
That said, the majority of exploits target systems which do not meet the “someone is actually trying to make this system secure” standard. If offense gets easier and defense remains at zero, offense becomes increasingly favored.
In most cases, if you want to be destructive you can spam unreliable attacks, less so if you want to build. this bodes poorly.
Regarding cybersecurity specifically, I have the unpopular opinion that it’s good actually that companies and govts are no longer able to protect their own privacy (or the privacy of their users).
What counts as “less concerned”?
Any event smaller than 10 million deaths is likely not going to register in my head as important enough for me to consider a career switch.