Backend/infrastructure engineer wandering into mechanistic interpretability and AI safety.
Interpretability, control, distributed systems, old poems, 4X games, experiments where nature gets to say “no,” and walls that don’t actually stop anyone. GitHub
Questions for the reader #1
The requirement of capturing evidence that can uniquely identify each ML workload (as in, every forward pass) is what I settled on. The reasoning behind this is that with this evidence, you can —in principle— screen for any governance objective. Do you think this is trying too much?
No and Yes. I believe the current trust environment calls for a ‘defense-in-depth’ strategy, and we need our perimeter defenses urgently. A near-term defense doesn’t need to conclusively verify a workload, it needs to produce enough signal to throw up a warning. [Baker et al.](https://arxiv.org/abs/2507.15916)’s six-layer framing is roughly what I have in mind—they don’t call it defense-in-depth, but the stack reads that way to me. Baker et al. call out workload classification as coarse and supplementary; I’d go farther and treat it as a backbone: our early warning system.
Therefore: No, pursuing granular evidence capture is not too ambitious *in the medium-long term*. It could become an instrumental trust lever as we build out ML infra, especially on specialized hardware or single-purpose training clusters. Per-forward-pass might not end up being the right level of granularity, but it’s a good starting point.
But Yes, it’s too much for right now: we need defenses that are cheap and broadly deployable. AI development increasingly exhibits race dynamics among actors whose capabilities and intentions are only partially visible. We see but through a glass darkly. [Gruetzemacher et al.](https://arxiv.org/abs/2410.03092)’s simulation work is suggestive here: across firms and states, actors repeatedly face incentives to compete even when coordination would leave them better off.
What verification goals do you think the field should actually focus on?
I think we should focus our effort on solutions that are near-term, broad and coarse. Make it real now; we can make it precise later. One concrete example: modality discrimination (training vs. inference vs. potentially training) from telemetry cloud providers already collect for operational reasons. It’s cheap, already instrumented, and the “potentially training” verdict is the point: it triggers inspection rather than requiring proof. [Seferis and Fist](https://ojs.aaai.org/index.php/AAAI/article/view/41127) influenced my thinking here.
Disclosure: I’m applying to work with Fowler as a mentee on a SPAR project related to interconnect-based training detection, so I have a minor stake in near-term, coarse observation work being worth doing.
Earnest offer: give me the prompts/transcripts (redacted in whatever way seems sensible) and I’ll go looking for the Doer in an open-weight model.
What would you want me to preregister as evidence for the Talker/Doer split before I start poking at activations?