Nice writeup! A few thoughts:
For OQ5, I’m quite excited about the Compartmentalization paper (but I’m also an author). My summary: from a frozen commit of model weights + inference code, any given inference output should be exactly reproducible from the frozen commit, a small amount of input data (~2MB for 1M tokens), and a small amount of compute. Any large training run will require exceeding the data and / or compute limit by 6 or more OOMs.
Example: If you wanted to do a training run on 1 trillion tokens (30-100x smaller than modern pretrains), then reproducing the final training checkpoint would require 6 more OOMs of both data input (the 1T tokens) and compute usage than a normal inference output. You could try to get around this by doing training batches that use the same number of tokens as a normal inference output. But, this requires persisting the training state, which isn’t included in the frozen commit. Reproducing any given training step would require the training state for Kimi-K3 (bf16 model weights + optimizer state), in the ballpark of 10-20 TB, which is 7 OOMs larger than a 1M token input. You can try shrink this with various tricks but there are a lot of OOMs to deal with.
For trusted replay, the determinism overhead isn’t bad. Deepseek uses batch-invariant kernel in prod training runs (https://arxiv.org/abs/2606.19348), where they report “minimal performance overhead”. Thinking Machines also does (https://thinkingmachines.ai/blog/interaction-models/), with less than 5% overhead. This also eliminates the prefill vs decode distinction, as a major motivation for labs to implement batch-invariance is to have bitwise reproducibility between inference and training (as seen in Thinking Machines post). However, this only enforces determinism on a fixed hardware selection.
If the prover and verifier have different hardware, the verifier won’t be able to exactly reproduce the prover’s results without significant overhead (GPT-5.6 guesses potentially 100-1000x, although I’m not sure) and significant engineering effort. But this would likely be an easy task for agents and random sampling is powerful, so it may be fine.
For OQ4: One advantage of trusted replay is that it would allow you to have an LLM in a box inspect traffic if you wanted to enforce treaty clauses like “no training” or “no AI research synthetic data generation” or “no bioweapons research”. Obviously there’s a risk of leaking sensitive info but an LLM in a box is a powerful affordance that we have never had before, and IMO is worth considering. Inference is a pretty simple process, and I would guess a good LLM could classify “inference code” vs “not-inference code”, even in adversarial settings.
This is quite speculative, but given that so far most algorithmic progress appears to be data-driven rather than architecture-driven, I’m not sure how important it is to avoid leaking architecture details. Although it’s obviously still a major issue.
Overhead from a deterministic stack is quite low and Deepseek and Thinking Machines both use it for production training runs. Deepseek reports “minimal performance overhead”, Thinking Machines reports 5%.
(More details in my below comment)