From what I gathered, determinism is the norm, not the exception. (I mean identical results within fixed conditions, not invariance across versions). HF transformers, vLLM, TensorRT-LLM. At least for inference. Deepseek v4′s inference code is fully deterministic, even batch-invariant.
This is quite straightforward to test as well: For a given inference setup, you can run the same prefill and decode runs repeatedly, even across different hardware. I did that for my paper: https://arxiv.org/abs/2606.00279v2
I found that only atomic operations with race conditions result in true non-determinism, and as you can read from Deepseek’s technical report, the incentive seems to be towards ensuring determinism, not breaking it. This is for RL stability, or training stability in general.
Reduction trees are pretty much the only thing you need to know for reproducibility, and that knowledge requires next to no disclosure of IP (when it comes to cuBLAS, which is what I worked with).
“I’m curious if this proposal necessitates forbidding multi-data center training.”
That’s a tricky edge case for the “all information in and out of a comput island is explainable”, for two reasons: One is the extreme information bandwidth, the other is that -as you explained- the links between datacenters would be hard to secure.
Since this is an edge-case and the size of upcoming megaclusters will probably eliminate the need for inter-datacenter pretraining, I expect this to not be an immediate dealbreaker.
If one really did insist on multi-datacenter training, you could still do the following: Each cluster is treated as an “island” whose in-out traffic must be fully explainable. Rather than capturing the whole traffic (which would be expensive), you can retrofit a setup that randomly samples from passively tapped optical fiber links. Basically a “panopticon” in which the prover can not predict which link will be sampled by the verifier, or when.
“(additionally, if multi-data center training isn’t allowable it also seems like you wouldn’t be able to transfer the completed model to a different data center, like one more suited for inference, after training)”
I am not quite following. Why would that be the case? Because the model has not been trained in a verified manner? Inference verification and training verification are different beasts. It probably makes sense to start with the former, and rely on post-hoc audits of models before training verification is ready.
Here is recent work on both, if you are interested: https://amododesign.com/notes/2026-06-23-verification-algorithms/