Thank you for writing this, it’s so great! I’ll respond line by line below:
OQ2: How can interconnect limits allow for video generation, which produces a lot of egress?
I agree that we care more about egress bandwidth (it seems like the bottleneck on using the monitored compute for training) but we might also care about a lot the ingress bandwidth, because the 1000x smaller dark compute could be used to do training while the monitored compute is being used for massive RL rollouts / data generation, and then using ingress video bandwidth (that we might want to allow to be very high for e.g., computer using agents or future robot-controlling-models) to upload new weights / low rank updates super quickly. So maybe additional open questions should be “how can interconnect limits allow for video inputs, which produce a lot of ingress?” This interacts with the memory wiping stuff below.
But I don’t understand this: why does it matter what a rack has in memory, provided every bit it emits is verified?
It’s mostly as defense in depth against spare compute being used for unverified stuff communicated out via side channels / covert channels (or tiny slippage in the verification to the extent we can’t get that to be literally 0). On this note, i think something like “How low can we get side channels bandwidth to be?” deserves to be its own Open Question.
Unfortunately, in an initial test, Amodo Design found that provably wiping a rack took ~24h and, if I’m reading their results correctly, left more than 100TB of memory unwiped.
A baseline proposal I was imagining (that I might not have included in the final writeup? oops if so) is that you have SSDs only allowed outside of the ‘inference-units’ (i.e., outside of the racks) so you avoid the unwiped 100TB, and then after memory wipes, you allow fast uploads from a bank of whitelisted model weights. If you did this alone I think you’d get to the 43 minutes according to their post? Then maybe you could do a hot reboot of the power going to the rack (to avoid the cold boot attacks that might persist volatile memory) instead of forced memorization and I would guess you could do even better? I’m surprised by the 24h latency though in their forced-memorization proposal, will look into it more.
Maybe another proposal would be to repurpose the backend networking to connect racks to a few ‘memory verifier’ servers, that the verifier controls, and uses to randomly choose a rack to refresh a rack by uploading a bunch of random noise to and checking it returns it perfectly. With 4 TB/s channels i’d guess this brings it down to ~minutes?
Plan A says that replay is used to “check the correctness of the packets”. I find this phrasing confusing: how are the packets produced by training any less “correct” than those produced by inference? My understanding is that what replay buys us depends on whether the replay server hardware is trusted by the verifier or by the prover (I assume that manufacturing a mutually-trusted replay server is impractical). I sketch the sorts of properties one could verify in each case below.
Two things here. (1) part of the proposal is for there to be a set of whitelisted inference-code + whitelisted set of model weights P*, so it’s easy to check that P \in P*. Sorry this wasn’t clear. I think this resolves OQ4? (2) Yes, I think we only need it to be unilaterally trusted by the verifier.
Agree that some cruxes here include (a) overhead from the deterministic stack, and (b) what % of samples need to be replayed for high detection probability (which as you say seems small due to power of sampling). Out of these, i think (a) might deserve to be its own Open Question?
The trust issues in this proposal seem quite bad to me: algorithmic secrets and sensitive user information are probably leaked.
Agree this is a potential problem (especially in versions of Plan A that are aiming for lower transparency than our default proposal, where this stuff is (mostly) supposed to be transparent anyway), so this also deserves to be an Open Question i think: “What maximum covert egress can we limit a verifier-trusted replay server to from the prover’s perspective?”
On the untrusted replay section, Daniel Reuter’s work indeed seems very promising here! Hopefully it works, I would be very excited to see the summary you said you might write up. I’m worried about how you would verify the “whitelisted model weights” part of the proposal though (i.e., to defend from a big RL / federated learning attack), but maybe we can just do that via the memory wipes + ingress/egress limits.
On ZKPs: [edits as of Aug 5 ’26] I think you are correct that the 1e6pretty high overhead might be fine because it benefits from the sampling math. This was a simple oversight on my side, because I didn’t connect until now that you can do the proofs post-hoc via challenges (i was previously naively imagining the prover needing to generate proofs for all tokens, rather than there being random challenges with e.g., 1/1e9 probability after the tokens are generated), thank you for surfacing this. Like you, i’m now also curious if anyone else can explain if this is missing something? I talked with some ZKP experts, and the problem with the 1e6 overhead is that you need to generate the proof for each token in context, which adds several OOMs (e.g., 4 OOMs with avg. 10k context). Apparently there are forthcoming solutions to this though.
Some further potential complications / questions i have on ZKPs are: - what is the required reproducibility in the ZKP case? e.g., is it much higher than the replay case because of the ZKP being more sensitive to tiny differences than the replay? - how robust can we make the ZKPs? Apparently there’s been a ‘track record of under-constrained circuits’ that let provers prove false statements silently? E.g., https://arxiv.org/abs/2402.15293, https://blog.trailofbits.com/2022/09/15/it-pays-to-be-circomspect/
Agree that some cruxes here include (a) overhead from the deterministic stack, and (b) what % of samples need to be replayed for high detection probability (which as you say seems small due to power of sampling). Out of these, i think (a) might deserve to be its own Open Question?
Overhead from a deterministic stack is quite low and Deepseek and Thinking Machines both use it for production training runs. Deepseek reports “minimal performance overhead”, Thinking Machines reports 5%.
Yeah these are promising signs, but I still think this deserves to be an Open Question going forwards in case it changes? Seems quite plausible to me that the overhead would be higher in the future (maybe even currently) on OpenAI / Anthropic production inference.
Talked with some ZKP experts. The problem is that you need to generate the proof for each token in the context, which adds several OOMs. Apparently there are forthcoming solutions to this though.
I talked with some ZKP experts, and the problem with the 1e6 overhead is that you need to generate the proof for each token in context, which adds several OOMs (e.g., 4 OOMs with avg. 10k context). Apparently there are forthcoming solutions to this though.
Why do we need to generate the proof for each token in the context? To be clear, I’m thinking of each decode step as a program that takes inputs = (kv_cache, input_tokens)and returns outputs = (new_keys_and_values, new_token). The prover publishes hashes of all keys and values. Then the verifier samples a decode step at random, and the prover must produce a ZKP that outputs = P(inputs).
Writing it this way, though, I see that I made an error in my previous estimate for the amount of covert communication the prover can perform without getting caught. I’d assumed that, for each fraudulent decode step, the prover can send one token’s worth of covert information out of the rack. I realize now that the correct amount of information is the size of outputs, which is dominated by new_keys_and_values. ChatGPT estimates new_keys_and_values is around 30kB for Kimi K3, which is ~4 OOMs larger than new_token. So, assuming 1e6 ZKP overhead as before, and 0.1% of compute set aside for producing ZKPs, the total covert communication volume is tens of terabytes (not gigabytes, as I claimed in the OP). Still doesn’t seem too bad to me.
Thank you for writing this, it’s so great! I’ll respond line by line below:
I agree that we care more about egress bandwidth (it seems like the bottleneck on using the monitored compute for training) but we might also care about a lot the ingress bandwidth, because the 1000x smaller dark compute could be used to do training while the monitored compute is being used for massive RL rollouts / data generation, and then using ingress video bandwidth (that we might want to allow to be very high for e.g., computer using agents or future robot-controlling-models) to upload new weights / low rank updates super quickly. So maybe additional open questions should be “how can interconnect limits allow for video inputs, which produce a lot of ingress?” This interacts with the memory wiping stuff below.
It’s mostly as defense in depth against spare compute being used for unverified stuff communicated out via side channels / covert channels (or tiny slippage in the verification to the extent we can’t get that to be literally 0). On this note, i think something like “How low can we get side channels bandwidth to be?” deserves to be its own Open Question.
A baseline proposal I was imagining (that I might not have included in the final writeup? oops if so) is that you have SSDs only allowed outside of the ‘inference-units’ (i.e., outside of the racks) so you avoid the unwiped 100TB, and then after memory wipes, you allow fast uploads from a bank of whitelisted model weights. If you did this alone I think you’d get to the 43 minutes according to their post? Then maybe you could do a hot reboot of the power going to the rack (to avoid the cold boot attacks that might persist volatile memory) instead of forced memorization and I would guess you could do even better? I’m surprised by the 24h latency though in their forced-memorization proposal, will look into it more.
Maybe another proposal would be to repurpose the backend networking to connect racks to a few ‘memory verifier’ servers, that the verifier controls, and uses to randomly choose a rack to refresh a rack by uploading a bunch of random noise to and checking it returns it perfectly. With 4 TB/s channels i’d guess this brings it down to ~minutes?
Two things here. (1) part of the proposal is for there to be a set of whitelisted inference-code + whitelisted set of model weights P*, so it’s easy to check that P \in P*. Sorry this wasn’t clear. I think this resolves OQ4?
(2) Yes, I think we only need it to be unilaterally trusted by the verifier.
Agree that some cruxes here include (a) overhead from the deterministic stack, and (b) what % of samples need to be replayed for high detection probability (which as you say seems small due to power of sampling). Out of these, i think (a) might deserve to be its own Open Question?
Agree this is a potential problem (especially in versions of Plan A that are aiming for lower transparency than our default proposal, where this stuff is (mostly) supposed to be transparent anyway), so this also deserves to be an Open Question i think: “What maximum covert egress can we limit a verifier-trusted replay server to from the prover’s perspective?”
On the untrusted replay section, Daniel Reuter’s work indeed seems very promising here! Hopefully it works, I would be very excited to see the summary you said you might write up. I’m worried about how you would verify the “whitelisted model weights” part of the proposal though (i.e., to defend from a big RL / federated learning attack), but maybe we can just do that via the memory wipes + ingress/egress limits.
On ZKPs: [edits as of Aug 5 ’26]
I think you are correct that
the 1e6pretty high overhead might be fine because it benefits from the sampling math. This was a simple oversight on my side, because I didn’t connect until now that you can do the proofs post-hoc via challenges (i was previously naively imagining the prover needing to generate proofs for all tokens, rather than there being random challenges with e.g., 1/1e9 probability after the tokens are generated), thank you for surfacing this.Like you, i’m now also curious if anyone else can explain if this is missing something?I talked with some ZKP experts, and the problem with the 1e6 overhead is that you need to generate the proof for each token in context, which adds several OOMs (e.g., 4 OOMs with avg. 10k context). Apparently there are forthcoming solutions to this though.
Some further potential complications / questions i have on ZKPs are:
- what is the required reproducibility in the ZKP case? e.g., is it much higher than the replay case because of the ZKP being more sensitive to tiny differences than the replay?
- how robust can we make the ZKPs? Apparently there’s been a ‘track record of under-constrained circuits’ that let provers prove false statements silently? E.g., https://arxiv.org/abs/2402.15293, https://blog.trailofbits.com/2022/09/15/it-pays-to-be-circomspect/
Overhead from a deterministic stack is quite low and Deepseek and Thinking Machines both use it for production training runs. Deepseek reports “minimal performance overhead”, Thinking Machines reports 5%.
(More details in my below comment)
Yeah these are promising signs, but I still think this deserves to be an Open Question going forwards in case it changes? Seems quite plausible to me that the overhead would be higher in the future (maybe even currently) on OpenAI / Anthropic production inference.
Talked with some ZKP experts. The problem is that you need to generate the proof for each token in the context, which adds several OOMs. Apparently there are forthcoming solutions to this though.
Why do we need to generate the proof for each token in the context? To be clear, I’m thinking of each decode step as a program that takes
inputs = (kv_cache, input_tokens)and returnsoutputs = (new_keys_and_values, new_token). The prover publishes hashes of all keys and values. Then the verifier samples a decode step at random, and the prover must produce a ZKP thatoutputs = P(inputs).Writing it this way, though, I see that I made an error in my previous estimate for the amount of covert communication the prover can perform without getting caught. I’d assumed that, for each fraudulent decode step, the prover can send one token’s worth of covert information out of the rack. I realize now that the correct amount of information is the size of
outputs, which is dominated bynew_keys_and_values. ChatGPT estimatesnew_keys_and_valuesis around 30kB for Kimi K3, which is ~4 OOMs larger thannew_token. So, assuming 1e6 ZKP overhead as before, and 0.1% of compute set aside for producing ZKPs, the total covert communication volume is tens of terabytes (not gigabytes, as I claimed in the OP). Still doesn’t seem too bad to me.