Re-execution is a much bigger ask than the draft implies. Bit-exact replay is a property the whole stack must hold all at once: the framework, math libraries, collective comms, the compiler’s autotuning, every custom kernel, the entire data pipeline. One nondeterministic component anywhere breaks it. Some ops have no deterministic implementation; much of the stack is closed-vendor code you can’t determinize yourself; and any version bump can silently break bit-exactness. So you either banish the nondeterminism and freeze the whole stack or commit to revalidating it forever. Determinizing one model’s inference is a demonstrated proof of concept. Requiring it across every training run, kernel, and pipeline on a fleet, permanently, is a stack-wide re-engineering nightmare, not just an enormous throughput hit. I’d expect that throttles the majority of real workloads, benign or otherwise.
That tax falls on benign and forbidden compute alike, so it doesn’t single out cheating. It just makes the monitored regime prohibitively slow and expensive for everyone. The consequence is a strong incentive to keep serious work off the monitored hardware. As others have noted, the scheme only governs compute that’s enrolled in the first place. If running under the monitor means giving up the tools and techniques high-performance computing is built on (or fully re-engineering it end-to-end), both benign and forbidden compute migrate to whatever capacity isn’t so severely constrained.
From what I gathered, determinism is the norm, not the exception. (I mean identical results within fixed conditions, not invariance across versions). HF transformers, vLLM, TensorRT-LLM. At least for inference. Deepseek v4′s inference code is fully deterministic, even batch-invariant.
This is quite straightforward to test as well: For a given inference setup, you can run the same prefill and decode runs repeatedly, even across different hardware. I did that for my paper: https://arxiv.org/abs/2606.00279v2
I found that only atomic operations with race conditions result in true non-determinism, and as you can read from Deepseek’s technical report, the incentive seems to be towards ensuring determinism, not breaking it. This is for RL stability, or training stability in general.
Reduction trees are pretty much the only thing you need to know for reproducibility, and that knowledge requires next to no disclosure of IP (when it comes to cuBLAS, which is what I worked with).
Conceded on deterministic inference. You know that layer better than I do, and it’s less of a burden than I thought.
Training is the harder half, and by your own description, a different beast. Determinism isn’t the default. The steps are stateful, so there’s no cheap sampled re-check to fall back on. But suppose that gets solved too. You still haven’t answered Fowler on false positives. The covert-adversary argument (detection × cost > gain, one hit is enough) only works if a flag means cheating. Here, the detector is inspector agents (5.2.3), which throw false positives.
That changes the cheater’s strategy. Rather than evade your samples, they can sit inside the noise band and let the false alarms teach your verifier to wave anomalies off. Which is the §2b attribution problem again, still the least-specified part of the proposal. I don’t think it belongs with the “what to do with a flag” politics you bracketed, either. The false-positive rate is a number the detector itself produces, and it limits what any escalation procedure could attribute, regardless of who runs it. The fallback to post-hoc audits on finished models is where the ambiguity is worst.
Capture looks like an engineering problem, at least. I don’t see what you’d build to fix attribution.
Re-execution is a much bigger ask than the draft implies. Bit-exact replay is a property the whole stack must hold all at once: the framework, math libraries, collective comms, the compiler’s autotuning, every custom kernel, the entire data pipeline. One nondeterministic component anywhere breaks it. Some ops have no deterministic implementation; much of the stack is closed-vendor code you can’t determinize yourself; and any version bump can silently break bit-exactness. So you either banish the nondeterminism and freeze the whole stack or commit to revalidating it forever. Determinizing one model’s inference is a demonstrated proof of concept. Requiring it across every training run, kernel, and pipeline on a fleet, permanently, is a stack-wide re-engineering nightmare, not just an enormous throughput hit. I’d expect that throttles the majority of real workloads, benign or otherwise.
That tax falls on benign and forbidden compute alike, so it doesn’t single out cheating. It just makes the monitored regime prohibitively slow and expensive for everyone. The consequence is a strong incentive to keep serious work off the monitored hardware. As others have noted, the scheme only governs compute that’s enrolled in the first place. If running under the monitor means giving up the tools and techniques high-performance computing is built on (or fully re-engineering it end-to-end), both benign and forbidden compute migrate to whatever capacity isn’t so severely constrained.
From what I gathered, determinism is the norm, not the exception. (I mean identical results within fixed conditions, not invariance across versions). HF transformers, vLLM, TensorRT-LLM. At least for inference. Deepseek v4′s inference code is fully deterministic, even batch-invariant.
This is quite straightforward to test as well: For a given inference setup, you can run the same prefill and decode runs repeatedly, even across different hardware. I did that for my paper: https://arxiv.org/abs/2606.00279v2
I found that only atomic operations with race conditions result in true non-determinism, and as you can read from Deepseek’s technical report, the incentive seems to be towards ensuring determinism, not breaking it. This is for RL stability, or training stability in general.
Reduction trees are pretty much the only thing you need to know for reproducibility, and that knowledge requires next to no disclosure of IP (when it comes to cuBLAS, which is what I worked with).
Conceded on deterministic inference. You know that layer better than I do, and it’s less of a burden than I thought.
Training is the harder half, and by your own description, a different beast. Determinism isn’t the default. The steps are stateful, so there’s no cheap sampled re-check to fall back on. But suppose that gets solved too. You still haven’t answered Fowler on false positives. The covert-adversary argument (detection × cost > gain, one hit is enough) only works if a flag means cheating. Here, the detector is inspector agents (5.2.3), which throw false positives.
That changes the cheater’s strategy. Rather than evade your samples, they can sit inside the noise band and let the false alarms teach your verifier to wave anomalies off. Which is the §2b attribution problem again, still the least-specified part of the proposal. I don’t think it belongs with the “what to do with a flag” politics you bracketed, either. The false-positive rate is a number the detector itself produces, and it limits what any escalation procedure could attribute, regardless of who runs it. The fallback to post-hoc audits on finished models is where the ambiguity is worst.
Capture looks like an engineering problem, at least. I don’t see what you’d build to fix attribution.