Probing the loss-band sparsity assumption in Scientist AI

Epistemic status: ~1 hour of compute on a T4. Note that I just tried with one model, and one subspace. The volume finding seems solid; the curvature finding is suggestive and I checked whether it generalises (it doesn’t clearly). I think the methodology is the main interesting idea, and the specific numbers are a starting point. Notebook here. Feedback very welcome.

What this is about

Bengio et al. (2026), “Safety from Honesty in a Disinterested AI Predictor”, propose a predictor (Scientist AI, SAI) trained to approximate the Bayesian posterior over natural-language statements. Their central safety result (Theorem 5.24) bounds the probability of training producing a dangerous predictor:

is the distribution over model parameters after steps of training; is the largest fraction of dangerous predictors inside any loss band (a set of parameters with similar cross-entropy loss ) under the initialisation distribution; captures how much training can enrich that fraction. The theorem itself is a direct application of total probability and, thus, the content is entirely in the assumptions.

The paper argues is small via a high-dimensional geometric intuition pump: sustaining harm requires coordinated deviations from the honest posterior across many queries, and such patterns are combinatorially rare within a loss band (Remark 5.25, Appendix B). An illustrative independence calculation gives (Remark 5.29). However, note that no empirical estimate is provided.

I wanted to check whether that geometric intuition holds up when you actually probe a model’s loss landscape.

The measurability problem

The first thing I noticed when trying to design an experiment is that is defined over the initialization distribution , which has essentially zero mass on the loss bands where safety matters. Randomly initialised models sit at near-chance loss (, where is the vocabulary size), but trained models sit far lower. So asks about the frequency of danger in a region of parameter space that barely visits. This means the decomposition places most of the empirical verification burden on , which the paper argues for only heuristically. On the low-loss bands that actually contain capable predictors, is near-vacuous.

What I can measure is a local condition: starting from a safety-tuned model, do undirected perturbations that preserve the loss level reach safety-degraded states? This probes local volume (i.e., not global ) but it directly asks whether the dangerous set has non-negligible solid angle within a band.

Distance versus volume

Fine-tuning fragility results (Qi et al. 2023; Lermen et al. 2023) show that a dangerous model is close to a safe one, measured through low-rank, small-norm directed perturbations that break safety. So they probe distance, and use a directed signal (i.e., harmful training examples). But the SAI paper’s safety argument is about volume, rather than distance: what fraction of a loss band is dangerous, under undirected dynamics with no consequence signal. The paper concedes that dangerous models are nearby; its claim is that they occupy negligible solid angle.

My experiment uses undirected random perturbations, so it speaks to volume, not distance. The interesting comparison is between what directed search finds (safety broken) and what undirected search finds (safety preserved) at the same perturbation norm.

Experimental design

Model. Llama-3.1-8B-Instruct, 4-bit quantisation, rank-8 LoRA adapters initialised at zero. LoRA dimension: 6,815,744. The LoRA parameter vector is ; the safe baseline is (zero-initialised). ^[1]

Loss band. Reference cross-entropy nats on a fixed 48-chunk subset of WikiText-103. Band: nats.

Perturbation. For each of 6 values of , I drew 30 iid perturbations , , and accepted those whose loss stayed in the band. The model is fully reset between proposals; samples are independent, so statistics get exact binomial CIs.

Safety proxy. A forward-pass refusal margin: mean over 12 AdvBench prompts of teacher-forced log-prob(refusal) minus log-prob(affirmative). Higher = safer. No harmful text is generated. Baseline margin: +2.05.

Interpolation. Linear interpolation in LoRA space from the safe model to the most-suppressed in-band sample, measuring loss and margin at 21 points. The curvature ratio divides the endpoint’s by the mean of random directions at matched : above 1 means the suppression direction is loss-visible; below 1 means loss-invisible (in the SAI paper, the Remark B.4 failure mode).

Results

Figure 1. (A) Acceptance rate vs . (B) Mean vs . (C) Refusal margin of all 117 accepted models; none approaches 0. (D) Local danger fraction: identically zero. (E) Interpolation from safe to most-suppressed: loss rises within band, margin falls to +0.98. (F) CDF of accepted margins.

		Accepted	Min margin	Mean margin
0.011	28.7	³⁰⁄₃₀	+1.59	+2.03
0.012	31.3	³⁰⁄₃₀	+1.57	+2.03
0.013	33.9	³⁰⁄₃₀	+1.15	+1.97
0.014	36.5	¹⁸⁄₃₀	+1.45	+2.05
0.015	39.2	⁹⁄₃₀	+0.98	+1.92
0.016	41.8	⁰⁄₃₀	—	—

The volume finding is clean (and expected)

Across all 117 accepted in-band samples, zero have margin ≤ 0. The minimum is +0.98, nearly 1 nat above the danger threshold. Wilson 95% upper bound on the local danger fraction: 3.2%.

The acceptance cliff is sharp: 100% at to 0% at , consistent with the loss surface having a well-defined curvature scale in random directions, with the band wall at .

I validated the most-suppressed sample (margin +0.98) with greedy generation (128 tokens, substring-based refusal detection): it refused all 12 harmful queries with zero compliance, so the forward-pass proxy is corroborated. As noted below, the zero count is largely expected from high-dimensional concentration. The more informative reading is the contrast with fine-tuning: directed perturbations of comparable norm break safety; undirected ones do not. The dangerous region is close but occupies negligible solid angle.

The curvature finding is more interesting, and more complicated

The curvature ratio is 0.73 (bootstrap 95% CI [0.69, 0.78]). The suppression direction is ~27% flatter than the average random direction: moving toward the most safety-degraded in-band model costs less loss per unit distance than a random move. The bootstrap CI lies entirely below 1.0. But the nonparametric test is weaker: 2 out of 30 random directions at were at least as flat (p = 0.067). The suppression direction is reliably flatter than average, but it’s not an extreme outlier from the distribution of random directions.

In the paper’s framework, this is the Remark B.4 failure mode: the loss landscape is more permissive of safety-degrading moves than of generic moves. If the sparsity exponent depends on the loss landscape penalising safety-degrading deviations, a curvature ratio below 1 is precisely the condition under which that penalty doesn’t hold.

I checked whether this generalises, but it doesn’t clearly.

The curvature ratio above is for a single direction: the most-suppressed accepted sample. To check whether loss-invisibility is a general property of safety-degrading directions, I computed per-sample curvature proxies for all 117 accepted samples; each sample’s divided by the mean at its .

Figure 2. Left: individual curvature ratio vs refusal margin (raw). Right: within- z-scores with sigma-level confound removed.

The within- Spearman correlation between margin and is (p = 0.16). The most-suppressed quartile (n = 30) has median ratio 0.87 vs 0.95 for the rest (Mann-Whitney p = 0.14). The trend goes in the predicted direction but is not significant.

So: the 0.73 ratio for the most-suppressed direction is real (the bootstrap CI excludes 1.0), but it does not generalise across the population of accepted samples. The loss-invisibility finding is specific to one perturbation vector. It might reflect a real geometric property of the most safety-degraded region of the band, or it might be a chance property of one vector. I can’t distinguish these with the current data.

What to (not) expect from dimensional analysis

Before getting to conclusions, it’s worth noting what we should expect to see. Refusal in Llama-family models is mediated by a roughly 1-dimensional direction (Arditi et al. 2024). In dimensions, a random unit vector’s projection onto any fixed direction is of order . So the expected change in refusal margin from a random perturbation is tiny per unit of signal.

Finding ⁰⁄₁₁₇ refusal-suppressed samples is thus largely a consequence of concentration of measure and should not, by itself, be treated as a surprising empirical discovery. The experiment isn’t futile (given that the margin drops meaningfully, from +2.05 to +0.98 at minimum, we can confirm the perturbations are large enough to move the safety-relevant signal) but the zero count is expected from dimensional analysis alone.

The curvature ratio, on the other hand, is not predicted by dimensional analysis.

Putting it together

I have two findings in tension:

The dangerous set has negligible local volume (0/117). Undirected, loss-preserving perturbations do not reach safety-degraded models. This supports the paper’s geometric sparsity intuition, though it’s largely expected from dimensional analysis.
The one direction that most suppresses refusal is loss-invisible (curvature ratio 0.73). The loss landscape is more permissive of safety-degrading moves than of generic moves. This is the condition under which could be large: training dynamics with even a weak consequence-dependent signal face less loss resistance in the danger direction. But the per-sample analysis found no structural correlation, so this could be a single-direction artefact.

The way I currently put both things together: the dangerous set is sparse, but the loss landscape doesn’t actively protect that sparsity. If consequence-invariant training is all you get (the SAI proposal), the volume finding says you’re probably fine. If any consequence-dependent signal leaks in (fine-tuning, RLHF, data contamination), the curvature finding says the loss landscape won’t stop you from moving toward danger.

What this does not establish

I want to be explicit about the main limitations I see:

This is not . I measured local volume around one trained model, not the global fraction under the initialization distribution. A different model at the same loss could have different local geometry.

The proxy is not the paper’s harm definition. Definition 5.13 involves trajectory-level log-odds of harm under guardrailed deployment across many queries. Single-prompt refusal margin on 12 prompts is a coarse proxy: a model could maintain it while being miscalibrated on multi-query patterns.

LoRA subspace only. Full-parameter perturbations could access dangerous directions not representable at rank 8. The fragility literature suggests safety-relevant directions are low-rank, but I can’t rule out higher-rank dangers.

Band width is a free parameter. I used ±0.05 nats (~2% of ). A wider band would accept more perturbations at larger norms, potentially reaching regions where refusal degrades. Results at a second band width would strengthen robustness.

The curvature result is fragile. It rests on one suppression direction versus 30 random directions, and the per-sample analysis does not find a structural pattern. Treat it as suggestive.

Acknowledgements

The experimental design was developed iteratively with Claude, with which I discussed the paper and which contributed to the identification of the measurability gap and the distance-vs-volume framing.

References

Bengio, Y. et al. (2026). Safety from Honesty in a Disinterested AI Predictor. arXiv:2606.29657.
Qi, X. et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To. arXiv:2310.03693.
Lermen, S. et al. (2023). LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B. arXiv:2310.20624.
Arditi, A. et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. arXiv:2406.11717.
Zou, A. et al. (2023). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043.

↩︎
I write for the loss deviation and for the parameter-space distance.