This is easily the most underrated AI security post of 2024, mostly because it points out that, unlike most files a defender has, AI weights are very, very large files, and that this means you can slow down exfiltration attempts enough to require physical attacks to be used.
The big question that will need to be answered is whether or not preventing exfiltration of model weights is positive EV, and I currently think it probably is, though there are reasonable arguments that state-proof or even state-resistant security worsens things via making the AI race more safe to engage in, and especially in worlds where long pauses are necessary, security could plausibly be negative, but there’s no way to get robustly positive actions on AI, so you have to use assumptions that could fail.
And at any rate, this is not relevant for the review at hand.
Ryan Greenblatt made a comment that reasoning models have made AI models smaller, but we now know that current RL/test time compute and reasoning was a one-time boost, and we will almost certainly see scaleups of AI models starting in 2026.
This becomes even more impactful if we believe that for the first AGIs, their files will be around the order of magnitude of human brain memory, which is 2.5 petabytes, which makes exfiltration much harder.
+9 for popularizing an idea so good that there’s a realistic chance we could jump from Security Level 2 to Security Level 4 with one idea, where we become resistant to state attacks (and if side effects turn out not to be a viable means of exfiltrating models, we could jump to Security Level 5, which requires defense against top nation states).
To be clear, I’m probably not (highly) conterfactual for other work in this area. As I note in this post:
While I expect that the sort of proposal I discuss here is well known, there are many specific details I discuss here which I haven’t seen discussed elsewhere. If you are reasonably familiar with this sort of proposal, consider just reading the “Summary of key considerations” section which summarizes the specific and somewhat non-obvious points I discuss in this post.
I think this idea has been independently invented many times; this post mostly adds additional consideration and popularization.
(TBC, I tenatively believe this post had a reasonably large impact, but through these mechanisms not by inventing the idea!)
This is easily the most underrated AI security post of 2024, mostly because it points out that, unlike most files a defender has, AI weights are very, very large files, and that this means you can slow down exfiltration attempts enough to require physical attacks to be used.
It also inspired follow-up work like Defending Against Model Weight Exfiltration Through Inference Verification to achieve the goal of preventing model exfiltration.
The big question that will need to be answered is whether or not preventing exfiltration of model weights is positive EV, and I currently think it probably is, though there are reasonable arguments that state-proof or even state-resistant security worsens things via making the AI race more safe to engage in, and especially in worlds where long pauses are necessary, security could plausibly be negative, but there’s no way to get robustly positive actions on AI, so you have to use assumptions that could fail.
And at any rate, this is not relevant for the review at hand.
Ryan Greenblatt made a comment that reasoning models have made AI models smaller, but we now know that current RL/test time compute and reasoning was a one-time boost, and we will almost certainly see scaleups of AI models starting in 2026.
This becomes even more impactful if we believe that for the first AGIs, their files will be around the order of magnitude of human brain memory, which is 2.5 petabytes, which makes exfiltration much harder.
+9 for popularizing an idea so good that there’s a realistic chance we could jump from Security Level 2 to Security Level 4 with one idea, where we become resistant to state attacks (and if side effects turn out not to be a viable means of exfiltrating models, we could jump to Security Level 5, which requires defense against top nation states).
To be clear, I’m probably not (highly) conterfactual for other work in this area. As I note in this post:
I think this idea has been independently invented many times; this post mostly adds additional consideration and popularization.
(TBC, I tenatively believe this post had a reasonably large impact, but through these mechanisms not by inventing the idea!)