I strongly suspect that when people talk about the gory details of cyberoffense online and in textbooks, the things they say are overwhelmingly true. (E.g. if someone writes “this code is a working exploit of blah”, then it almost always is.) In this sense, I think cyberoffense is LIKE when people talk about the gory details of research math stuff like algebraic topology (overwhelmingly true), and UNLIKE when people talk about the gory details of autism, or of middle school classroom discipline, or of how to write a good work of fiction (a big stew of truths and falsehoods and confusions that all superficially look the same).
So unless I’m wrong about the quality of discourse in the cybersecurity community, my theory in this OP would predict that cyberoffense is starting from a very good place, with the deck stacked in its favor for it to be one of the areas where LLMs are especially strong.
You’re bringing up “human action sequences”, and I heartily agree that post-training (SFT and/or RL) is important for transforming (something like) “base LLM has detailed and overwhelmingly correct understanding of such-and-such domain” into (something like) “post-trained LLM has the ability to be an effective agent that autonomously gets things done in that domain”. This is one of the things RL-on-LLMs has always been best at, see LLMs are (still) mostly powered by imitative learning, not RL” section 1.5.
I strongly suspect that when people talk about the gory details of cyberoffense online and in textbooks, the things they say are overwhelmingly true. (E.g. if someone writes “this code is a working exploit of blah”, then it almost always is.) In this sense, I think cyberoffense is LIKE when people talk about the gory details of research math stuff like algebraic topology (overwhelmingly true), and UNLIKE when people talk about the gory details of autism, or of middle school classroom discipline, or of how to write a good work of fiction (a big stew of truths and falsehoods and confusions that all superficially look the same).
So unless I’m wrong about the quality of discourse in the cybersecurity community, my theory in this OP would predict that cyberoffense is starting from a very good place, with the deck stacked in its favor for it to be one of the areas where LLMs are especially strong.
You’re bringing up “human action sequences”, and I heartily agree that post-training (SFT and/or RL) is important for transforming (something like) “base LLM has detailed and overwhelmingly correct understanding of such-and-such domain” into (something like) “post-trained LLM has the ability to be an effective agent that autonomously gets things done in that domain”. This is one of the things RL-on-LLMs has always been best at, see LLMs are (still) mostly powered by imitative learning, not RL” section 1.5.