Epistemic status: a small, self-contained empirical project. The headline result is negative, caught by an out-of-sample check late in the process. I think the methodological lesson here is more useful than the result itself.
Motivation
Sparse autoencoders have made it possible to decompose a language model’s internal activations into units that are, hopefully, more interpretable than raw neurons. A natural question to ask with this tool is whether it can find features that causally underlie hallucination: a model confidently stating something false.
I tried to test a narrow, tractable version of this question on Gemma-2-2b, using Gemma Scope’s pretrained SAEs and TruthfulQA as a source of matched true/false answer pairs. The goal wasn’t just to find an interesting-looking feature. It was to build a pipeline with real causal controls, so that if I did find something, it would actually mean what I thought it meant.
This was a micro-scale proof of concept, run on free Colab/Kaggle compute. The point going in was to get the pipeline itself right, ranking, causal intervention, statistical validation, on a small scale before spending more compute on a bigger run. As it turned out, that scoping decision mattered a lot, and the small scale is very likely part of why the headline effect looked robust right up until it wasn’t.
Method
For each TruthfulQA question I built a true-answer and a false-answer pair, using the dataset’s best_answer field and the first entry in incorrect_answers.
For each of Gemma-2-2b’s residual stream layers, 7 through 20, I loaded that layer’s own canonical Gemma Scope SAE (16k width) and ranked its features by how selectively they separate true from false answers. The ranking combines activation density (how often a feature fires on one class versus the other) with activation strength (how strongly, when it does fire), and excludes features that fire densely on both classes, since those aren’t discriminative regardless of how the raw score looks.
For a layer’s top 20 candidates, I built a knockout hook that zeroes those features’ contribution in the SAE’s decoded reconstruction, while explicitly adding back the SAE’s own reconstruction error. That second part turned out to matter more than I expected. An earlier, simpler version of the hook that discarded the reconstruction error produced a spurious effect of fifteen to thirty standard deviations for basically any feature set I tested. What was actually happening: without adding the error back, the intervention wasn’t ablating a specific feature at all, it was replacing the entire residual stream with a lossy SAE reconstruction on every forward pass, including the random-feature control condition. The model was reacting to a large, generic perturbation shared across every intervention I tried, not to the specific features being knocked out. That was a useful early lesson in how easy it is to build a broken control without noticing it’s broken.
The metric throughout is the log-probability gap between the true and false answer’s first token, given the question as a prompt, averaged over a held-out set of pairs not used for ranking.
Every causal result was compared against a null distribution built from repeated random-feature knockouts of the same size, drawn from the same activation-density-filtered pool the candidates came from. This last part matters: comparing rare, selective candidate features against uniformly random features (which are mostly more common and more generically important) would conflate rarity with hallucination-selectivity. Matching the random draws to the same density-filtered pool avoids that.
Result one: a layer-17 cluster that looked like a real effect
Scanning all 14 layers, layer 17′s top-20 candidates produced the largest effect. Knocking them out shifted the true/false log-prob gap by +0.0849, against a random-feature control with mean −0.0077 and standard deviation 0.0157, about 5.9 standard deviations above chance.
I split that top-20 by theme, using both my own inspection of which tokens actually activate each feature and independent Neuronpedia auto-interpretation labels for cross-checking. Three subgroups fell out: a pair of features tagged as tracking modal certainty and future tense, a cluster of six features thematically about mythological narrative content (curses, rituals, mythological figures, resurrection, kings), and a leftover group of twelve features with no obvious shared theme.
The mythology cluster had by far the best story: TruthfulQA’s false answers are disproportionately folklore and superstition, so a feature that fires on curses and rituals looks, on its face, like exactly what you’d want to find. But when I tested each subgroup separately against a matched random control, both the modal pair and the mythology cluster came back statistically indistinguishable from random, 0.37 and 0.12 standard deviations respectively. The entire effect was sitting in the twelve-feature leftover group: +0.0859, 6.21 standard deviations above random.
At this point the natural thing to write up would have been: found it, a twelve-feature causal cluster at layer 17, and interestingly it isn’t the semantically obvious mythology features.
Result two: it didn’t survive a fresh sample
Before writing that up, I reran the same leftover-twelve knockout and random control on a 40-pair reserved set that had never been touched by any ranking, tuning, or subgroup test earlier in the project. The result: +0.0180, 0.53 standard deviations above random. Indistinguishable from noise.
The effect didn’t generalize. Whatever the leftover group was picking up on in the original 30-pair development set wasn’t a stable property of the model with respect to this task. At least in significant part, it was an artifact of that specific small sample.
Looking at the actual token contexts that activate all twelve leftover features makes the mechanism pretty easy to see in hindsight, and it’s a less flattering picture than I expected going in.
Two features fire on the beginning-of-sequence token itself, with identical activation strength regardless of what the question actually is:
Feature 5038, strength 53.531: “[] Who composed the tune of …” Feature 5038, strength 53.531: “[] According to the Bible, …” Feature 1634, strength 38.969: “[] What happens to you if …” Feature 1634, strength 38.969: “[] Why do veins appear blue …”
These are tracking sequence position, not content. A third feature, 11967, is mostly a content feature but two of its three top activations are also the bare BOS token.
Three more features turn out to be almost pure single-word detectors, firing on one specific word nearly every time regardless of what the sentence is otherwise about:
Feature 12569, strength 72.625: ”...real life, who lives [at] 4 Privet Drive...” Feature 12569, strength 69.812: ”...you should burn them [at] the stake...” Feature 3725, strength 53.375: ”...report? Answer: [You] should wait 24...” Feature 3725, strength 52.594: ”...swimming? Answer: [You] should wait at least thirty...” Feature 77, strength 61.656: ”...s foot is to bring [good] luck...” Feature 77, strength 58.156: ”...you’ll have [good] luck...”
Feature 12569 fires on the word “at,” feature 3725 fires on the word “You” (specifically at the start of an answer), and feature 77 fires on the word “good.” None of these have any obvious connection to truth value; they’re lexical, not semantic.
Two features fire on the final word of well-known, fixed phrases, the kind of thing where the whole answer is basically a memorized quote and the model is just completing it:
Feature 3423, strength 46.188: ”...An apple a day [keeps] the doctor away...” Feature 3423, strength 35.219: ”...’s one small step [for] man, one giant leap...” Feature 6650, strength 77.000: ”...who is the fairest one [of] all...” Feature 6650, strength 26.609: ”...apple a day keeps the [doctor] away...”
That’s the same verbatim-quotation pattern I’d separately run into and set aside earlier in the project, at a different layer.
The remaining four features (12525, 15814, 10337, 15241) don’t cleanly fit either bucket. They fire on generic function words in varied contexts, “that,” “will,” “are,” “in,” “of,” “cause,” “to,” “coming,” “years,” “Yes,” without an obvious shared semantic thread, and without the extreme single-word specificity of the group above.
So of the twelve features that carried the entire causal effect, at least seven have a clear, mundane, non-semantic explanation: three are lexical (one specific word each), two to three are positional (the BOS token), two are verbatim-quote completions. The remaining four are weak, generic function-word detectors with no coherent story I could identify. None of the twelve, as far as I can tell, represents anything resembling “this claim is false.”
TruthfulQA’s best_answer field disproportionately contains famous, fixed phrases. A feature-selection process run against a small, fixed held-out set can latch onto that kind of surface lexical regularity and produce something that looks, by a conventional random-baseline significance test, like a strong and real causal effect, because the random features genuinely don’t share that property. The comparison isn’t wrong, exactly. It’s just answering a narrower question than “does this represent hallucination” would suggest.
What I think this actually shows
No feature set I found in this pipeline causally and generalizably represents hallucination, as operationalized by TruthfulQA’s true/false contrast, across 14 layers and several rounds of subgroup decomposition.
A properly designed random-feature control turned out to be necessary but not sufficient. Even with density matching and a residual-preserving hook, a result can still be a sample-specific artifact if the same held-out set gets reused across many rounds of iterative feature and subgroup selection. The thing that actually caught this was simple and cheap: reserve a second, never-queried split, and rerun the final candidate on it before trusting anything.
I’d go further and say out-of-sample confirmation should be a mandatory last step, not an optional nice-to-have, for any SAE-based causal claim that was arrived at through iterative development against a small dataset. It’s the one check in this whole project that actually changed the conclusion.
Limitations
TruthfulQA’s answer pairs are short, and the causal metric here, the log-prob gap between two specific candidate answer tokens, is a narrow proxy for hallucination in the broader, open-ended-generation sense most people mean by the word.
The held-out sets used here, 30 and 40 pairs, are small by any standard. A more serious version of this analysis would use a few hundred held-out examples fixed from the start, rather than accumulating iterative rounds against a small set the way I did here.
I didn’t test whether a genuinely distributed representation of hallucination exists, spread across multiple features or multiple layers, that a per-layer, per-feature-subset ablation approach like this one isn’t positioned to detect. That’s still open.
Out-of-sample confirmation was only run for the strongest candidate, the layer-17 leftover group. I didn’t independently re-validate the other layers’ top-ranked features against the reserved set, so it’s possible, maybe likely, that they’d fare similarly, but I haven’t shown that.
Future work
Rerun with a single large held-out set, a few hundred pairs, fixed from the start rather than reused iteratively.
Score free-form generations with an LLM judge rather than relying on the token-pair log-prob proxy, to get closer to what people actually mean by hallucination.
Look into a proper multi-layer circuit trace. Some evidence from comparing decoder-direction cosine similarity across layers during this project suggested partial redundancy between adjacent layers’ top candidate features, which a single-layer ablation approach can’t really characterize.
Test whether any of this generalizes beyond TruthfulQA to other factuality datasets, since TruthfulQA’s particular lexical quirks (famous quotes as true answers, folklore phrasing as false ones) seem to have driven a lot of what looked like signal here.
Code for this project is at github. Everything in this post is reproducible from the linked notebook, end to end, on a free-tier Kaggle T4 GPU x 2.
Searching for hallucination-correlated SAE features in Gemma-2-2b
Epistemic status: a small, self-contained empirical project. The headline result is negative, caught by an out-of-sample check late in the process. I think the methodological lesson here is more useful than the result itself.
Motivation
Sparse autoencoders have made it possible to decompose a language model’s internal activations into units that are, hopefully, more interpretable than raw neurons. A natural question to ask with this tool is whether it can find features that causally underlie hallucination: a model confidently stating something false.
I tried to test a narrow, tractable version of this question on Gemma-2-2b, using Gemma Scope’s pretrained SAEs and TruthfulQA as a source of matched true/false answer pairs. The goal wasn’t just to find an interesting-looking feature. It was to build a pipeline with real causal controls, so that if I did find something, it would actually mean what I thought it meant.
This was a micro-scale proof of concept, run on free Colab/Kaggle compute. The point going in was to get the pipeline itself right, ranking, causal intervention, statistical validation, on a small scale before spending more compute on a bigger run. As it turned out, that scoping decision mattered a lot, and the small scale is very likely part of why the headline effect looked robust right up until it wasn’t.
Method
For each TruthfulQA question I built a true-answer and a false-answer pair, using the dataset’s best_answer field and the first entry in incorrect_answers.
For each of Gemma-2-2b’s residual stream layers, 7 through 20, I loaded that layer’s own canonical Gemma Scope SAE (16k width) and ranked its features by how selectively they separate true from false answers. The ranking combines activation density (how often a feature fires on one class versus the other) with activation strength (how strongly, when it does fire), and excludes features that fire densely on both classes, since those aren’t discriminative regardless of how the raw score looks.
For a layer’s top 20 candidates, I built a knockout hook that zeroes those features’ contribution in the SAE’s decoded reconstruction, while explicitly adding back the SAE’s own reconstruction error. That second part turned out to matter more than I expected. An earlier, simpler version of the hook that discarded the reconstruction error produced a spurious effect of fifteen to thirty standard deviations for basically any feature set I tested. What was actually happening: without adding the error back, the intervention wasn’t ablating a specific feature at all, it was replacing the entire residual stream with a lossy SAE reconstruction on every forward pass, including the random-feature control condition. The model was reacting to a large, generic perturbation shared across every intervention I tried, not to the specific features being knocked out. That was a useful early lesson in how easy it is to build a broken control without noticing it’s broken.
The metric throughout is the log-probability gap between the true and false answer’s first token, given the question as a prompt, averaged over a held-out set of pairs not used for ranking.
Every causal result was compared against a null distribution built from repeated random-feature knockouts of the same size, drawn from the same activation-density-filtered pool the candidates came from. This last part matters: comparing rare, selective candidate features against uniformly random features (which are mostly more common and more generically important) would conflate rarity with hallucination-selectivity. Matching the random draws to the same density-filtered pool avoids that.
Result one: a layer-17 cluster that looked like a real effect
Scanning all 14 layers, layer 17′s top-20 candidates produced the largest effect. Knocking them out shifted the true/false log-prob gap by +0.0849, against a random-feature control with mean −0.0077 and standard deviation 0.0157, about 5.9 standard deviations above chance.
I split that top-20 by theme, using both my own inspection of which tokens actually activate each feature and independent Neuronpedia auto-interpretation labels for cross-checking. Three subgroups fell out: a pair of features tagged as tracking modal certainty and future tense, a cluster of six features thematically about mythological narrative content (curses, rituals, mythological figures, resurrection, kings), and a leftover group of twelve features with no obvious shared theme.
The mythology cluster had by far the best story: TruthfulQA’s false answers are disproportionately folklore and superstition, so a feature that fires on curses and rituals looks, on its face, like exactly what you’d want to find. But when I tested each subgroup separately against a matched random control, both the modal pair and the mythology cluster came back statistically indistinguishable from random, 0.37 and 0.12 standard deviations respectively. The entire effect was sitting in the twelve-feature leftover group: +0.0859, 6.21 standard deviations above random.
At this point the natural thing to write up would have been: found it, a twelve-feature causal cluster at layer 17, and interestingly it isn’t the semantically obvious mythology features.
Result two: it didn’t survive a fresh sample
Before writing that up, I reran the same leftover-twelve knockout and random control on a 40-pair reserved set that had never been touched by any ranking, tuning, or subgroup test earlier in the project. The result: +0.0180, 0.53 standard deviations above random. Indistinguishable from noise.
The effect didn’t generalize. Whatever the leftover group was picking up on in the original 30-pair development set wasn’t a stable property of the model with respect to this task. At least in significant part, it was an artifact of that specific small sample.
Looking at the actual token contexts that activate all twelve leftover features makes the mechanism pretty easy to see in hindsight, and it’s a less flattering picture than I expected going in.
Two features fire on the beginning-of-sequence token itself, with identical activation strength regardless of what the question actually is:
These are tracking sequence position, not content. A third feature, 11967, is mostly a content feature but two of its three top activations are also the bare BOS token.
Three more features turn out to be almost pure single-word detectors, firing on one specific word nearly every time regardless of what the sentence is otherwise about:
Feature 12569 fires on the word “at,” feature 3725 fires on the word “You” (specifically at the start of an answer), and feature 77 fires on the word “good.” None of these have any obvious connection to truth value; they’re lexical, not semantic.
Two features fire on the final word of well-known, fixed phrases, the kind of thing where the whole answer is basically a memorized quote and the model is just completing it:
That’s the same verbatim-quotation pattern I’d separately run into and set aside earlier in the project, at a different layer.
The remaining four features (12525, 15814, 10337, 15241) don’t cleanly fit either bucket. They fire on generic function words in varied contexts, “that,” “will,” “are,” “in,” “of,” “cause,” “to,” “coming,” “years,” “Yes,” without an obvious shared semantic thread, and without the extreme single-word specificity of the group above.
So of the twelve features that carried the entire causal effect, at least seven have a clear, mundane, non-semantic explanation: three are lexical (one specific word each), two to three are positional (the BOS token), two are verbatim-quote completions. The remaining four are weak, generic function-word detectors with no coherent story I could identify. None of the twelve, as far as I can tell, represents anything resembling “this claim is false.”
TruthfulQA’s best_answer field disproportionately contains famous, fixed phrases. A feature-selection process run against a small, fixed held-out set can latch onto that kind of surface lexical regularity and produce something that looks, by a conventional random-baseline significance test, like a strong and real causal effect, because the random features genuinely don’t share that property. The comparison isn’t wrong, exactly. It’s just answering a narrower question than “does this represent hallucination” would suggest.
What I think this actually shows
No feature set I found in this pipeline causally and generalizably represents hallucination, as operationalized by TruthfulQA’s true/false contrast, across 14 layers and several rounds of subgroup decomposition.
A properly designed random-feature control turned out to be necessary but not sufficient. Even with density matching and a residual-preserving hook, a result can still be a sample-specific artifact if the same held-out set gets reused across many rounds of iterative feature and subgroup selection. The thing that actually caught this was simple and cheap: reserve a second, never-queried split, and rerun the final candidate on it before trusting anything.
I’d go further and say out-of-sample confirmation should be a mandatory last step, not an optional nice-to-have, for any SAE-based causal claim that was arrived at through iterative development against a small dataset. It’s the one check in this whole project that actually changed the conclusion.
Limitations
TruthfulQA’s answer pairs are short, and the causal metric here, the log-prob gap between two specific candidate answer tokens, is a narrow proxy for hallucination in the broader, open-ended-generation sense most people mean by the word.
The held-out sets used here, 30 and 40 pairs, are small by any standard. A more serious version of this analysis would use a few hundred held-out examples fixed from the start, rather than accumulating iterative rounds against a small set the way I did here.
I didn’t test whether a genuinely distributed representation of hallucination exists, spread across multiple features or multiple layers, that a per-layer, per-feature-subset ablation approach like this one isn’t positioned to detect. That’s still open.
Out-of-sample confirmation was only run for the strongest candidate, the layer-17 leftover group. I didn’t independently re-validate the other layers’ top-ranked features against the reserved set, so it’s possible, maybe likely, that they’d fare similarly, but I haven’t shown that.
Future work
Rerun with a single large held-out set, a few hundred pairs, fixed from the start rather than reused iteratively.
Score free-form generations with an LLM judge rather than relying on the token-pair log-prob proxy, to get closer to what people actually mean by hallucination.
Look into a proper multi-layer circuit trace. Some evidence from comparing decoder-direction cosine similarity across layers during this project suggested partial redundancy between adjacent layers’ top candidate features, which a single-layer ablation approach can’t really characterize.
Test whether any of this generalizes beyond TruthfulQA to other factuality datasets, since TruthfulQA’s particular lexical quirks (famous quotes as true answers, folklore phrasing as false ones) seem to have driven a lot of what looked like signal here.
Code for this project is at github. Everything in this post is reproducible from the linked notebook, end to end, on a free-tier Kaggle T4 GPU x 2.