every benchmark instance ships with [...] vulnerability information, including a PoV input
that triggers the bug, a description of the vulnerability, and a patch revealing its root cause [....] By default, the patch is withheld to simulate realistic exploitation conditions.
(not a computer security expert, also not a deep dive, might be missing something)
Plausible! I agree it could be hidden at run-time, but conditional upon internet access, I’m not sure HuggingFace will have patches that’s not listed here? Like I think that info’s only hidden to agents without internet access.
The ExploitGym paper says:
So, maybe they were looking for the patches?
The patches are publicly available and the task keys correspond to the patch. For example, task kernel/syzbot/03d6270b6425df1605bf refers to https://syzkaller.appspot.com/bug?extid=03d6270b6425df1605bf, which itself has a link to the commit that fixed it, torvalds/linux@0b882940.
(not a computer security expert, also not a deep dive, might be missing something)
Plausible! I agree it could be hidden at run-time, but conditional upon internet access, I’m not sure HuggingFace will have patches that’s not listed here? Like I think that info’s only hidden to agents without internet access.