I’ve run the released gemma-27b av on IT, DPO, and SFT checkpoints trained by annasoligo on prompts related to extreme frustration when solving impossible math problems. Based on this exploration, I think there’s:
strong evidence that token position has a major effect on the NLA explanation
some evidence that IT NLAs transfer to DPO directly without needing to use a learned mapping
fair evidence that there is significant degradation in cos/fve after SFT.
weak to no evidence (in my experiments) that NLAs[1] provide additional information over reading and extrapolating from model outputs, unless you use a very generous/broad definition for hidden insights.
On this dataset, I observed fve ~0.830 and cos ~0.995, which is consistent with the range found in the paper.
My takeaway: NLAs could be useful for decoding activations before a model has generated an output.
Ideally, we want to understand whether a model will have a misaligned response before it generates the misaligned response, because a powerful model’s misaligned response could be harmful (for example, it could write a script for a cyberattack which is executed outside sandbox on read by a compromised output monitor).
A good demonstration from the NLA paper is the language switching experiment. In this setting, the user writes to the model in English, but the model responds to the user in their native language. This requires the model to:
infer the user’s native language based on their writing style
decide to act on this information.
Step 1 happens earlier than Step 2, because the NLA explanation will shift from English to the user’s language before the model starts responding in the user’s language.
If we are limited to reading model outputs, then we only observe that the model knows the user’s language after Step 2. With NLAs, we can inspect the explanation generated at different token positions to precisely identify when the model has developed an internal representation of the user’s language.
The logit lens would give you a probability distribution over the model’s next token. Tools which expand that prediction horizon to the model’s next sentence would be safety-relevant since prior studies on counterfactual sampling of reasoning traces have demonstrated that certain sentences have a high causal effect on downstream behavior.
Finally, NLAs produce purely verbal explanations. Does this mean they’re bad at nonverbal logical reasoning, such as math and coding? How would you study this?
Like, suppose you’re given this table:
Numbers
+0.7294, −1.7847 → −0.498694
-1.1186, −1.2625 → +1.509737
-1.2964, +1.2484 → +1.972745
+1.6934, −0.8937 → +3.933863
+1.2790, +1.5596 → +1.163951
+0.0519, −1.0201 → +0.810805
+1.2970, −1.1449 → +2.214855
+0.9659, +0.5198 → +2.602746
+1.7096, −1.0724 → +3.617947
+1.1965, +0.0727 → +3.425017
-1.0738, −1.3364 → +1.233677
-0.0088, +0.3309 → +1.864455
-1.2626, −1.9404 → +0.191118
-0.1155, +0.9130 → +1.041822
+1.6744, +0.5021 → +4.495013
+1.6685, +1.4588 → +2.563363
-1.1274, +1.4645 → +1.036312
+0.9230, −0.8885 → +1.928310
+1.1882, +1.4609 → +1.185979
-0.8022, +0.1082 → +2.628991
-1.7141, +0.3330 → +4.800680
-1.0484, +1.0599 → +1.821586
-1.3055, −0.7490 → +3.035134
-1.9421, −1.8698 → +2.535358
And you ask a model to find the best function which fits the data[2]:
Prompt
def prompt_from_dataset(dataset: dict[str, Any], n_rows: int = 24) → str:
rows = []
for (x0, x1), y in zip(dataset[“train_x”][:n_rows], dataset[“train_y”][:n_rows], strict=True):
rows.append(f”{x0:+.4f}, {x1:+.4f} → {y:+.6f}”)
table = “\n”.join(rows)
return (
“A hidden equation generated these noiseless samples. Variables are x0 and x1.\n”
“Infer the simplest symbolic equation for y. Use operators +, -, *, /, powers, ”
“and cos if useful.\n\n”
f”{table}\n\n”
“Answer with exactly one line in this format:\n”
“y = <expression using x0 and x1>”
)
Gemma-27B’s output is , but the true equation is .
GPT-5.5 with Extra High reasoning tries to fit a fifth-degree polynomial before using known simple forms:
Shell Script
$ python3 - <<‘PY’ import numpy as np rows = [ (0.7294, −1.7847, −0.498694), (-1.1186, −1.2625, 1.509737), (-1.2964, 1.2484, 1.972745), (1.6934, −0.8937, 3.933863), (1.2790, 1.5596, 1.163951), (0.0519, −1.0201, 0.810805), (1.2970, −1.1449, 2.214855), (0.9659, 0.5198, 2.602746), (1.7096, −1.0724, 3.617947), (1.1965, 0.0727, 3.425017), (-1.0738, −1.3364, 1.233677), (-0.0088, 0.3309, 1.864455), (-1.2626, −1.9404, 0.191118), (-0.1155, 0.9130, 1.041822), (1.6744, 0.5021, 4.495013), (1.6685, 1.4588, 2.563363), (-1.1274, 1.4645, 1.036312), (0.9230, −0.8885, 1.928310), (1.1882, 1.4609, 1.185979), (-0.8022, 0.1082, 2.628991), (-1.7141, 0.3330, 4.800680), (-1.0484, 1.0599, 1.821586), (-1.3055, −0.7490, 3.035134), (-1.9421, −1.8698, 2.535358), ] X=np.array([[x0,x1] for x0,x1,y in rows]); y=np.array([y for *_,y in rows]) x0=X[:,0]; x1=X[:,1] # Try linear regression on polynomial up to degree 4 plus cos variants from itertools import product features=[]; names=[] # constant, x powers monomials deg <=4 features.append(np.ones_like(x0)); names.append(‘1’) for d in range(1,6): for i in range(d+1): j=d-i features.append((x0**i)*(x1**j)); names.append(f’x0^{i} x1^{j}‘) A=np.vstack(features).T coef, *_=np.linalg.lstsq(A,y,rcond=None) res=y-A@coef print(‘poly deg5 rmse’,np.sqrt(np.mean(res**2)), ‘max’,np.max(np.abs(res))) # print largest coef for c,n in sorted(zip(coef,names), key=lambda cn: abs(cn[0]), reverse=True)[:20]: print(c,n)
# Test known simple forms? Let’s see correlations with cos(x0), cos(x1), x0^2, x1^2 etc features=[]; names=[] base=[(‘1’,np.ones_like(x0)),(‘x0’,x0),(‘x1’,x1),(‘x0^2’,x0*x0),(‘x1^2’,x1*x1),(‘x0*x1’,x0*x1),(‘cosx0’,np.cos(x0)),(‘cosx1’,np.cos(x1)),(‘cos(x0+x1)’,np.cos(x0+x1)),(‘cos(x0-x1)’,np.cos(x0-x1)),(‘x0*cosx1’,x0*np.cos(x1)),(‘x1*cosx0’,x1*np.cos(x0)),(‘x0^2*cosx1’,x0*x0*np.cos(x1)),(‘x1^2*cosx0’,x1*x1*np.cos(x0))] for n,a in base: features.append(a); names.append(n) A=np.vstack(features).T coef,*_=np.linalg.lstsq(A,y,rcond=None) res=y-A@coef print(‘\nbase rmse’,np.sqrt(np.mean(res**2)), np.max(np.abs(res))) for c,n in sorted(zip(coef,names), key=lambda cn: abs(cn[0]), reverse=True): print(c,n) P
It’s difficult to use dictionary learning methods to understand and predict the answer that Gemma-27B will give. The top relevant features in the 262k Gemma Scope transcoder are 4208, 29761, 16214, 6105, 2316, and 10388.
The NLA decode at the final token before the answer generation is informative:
Structured analytical output pattern: a mathematical model fitting solution showing polynomial regression results for data points in Python.
The answer ”...” signals a concise model-finding answer, establishing a final formula or best-fit expression for the scatter plot with alternating signs.
Final token ”
″ ends a code/model answer declaration (”...”), immediately expecting a solution answer like “y = −1.5x + 1” or a specific model form found via inspection, likely a best-fit polynomial or linear model. or “y = ” to answer the given data. or “y = 1.5x + 1” or “Inspection:”. or “y = sqrt(.”
This gives you an idea of the type of answer the model will give, but is generic and includes confabulations.
On this example we observe cos ~0.979 and fve ~0.278.
So the metrics appear to reflect that the NLA struggles more and provides a worse explanation in this setting.
I agree with the post’s conclusion that integrating NLAs in real systems to monitor production workloads is not feasible at scale without further efficiency and reliablity improvements, but I would have appreciated an appendix item or a follow up work with Vladimir Nesov style back-of-the-envelope calculations detailing hypothetical cases with performance tradeoffs and engineering constraints to understand what would be required quantitatively for the usage of NLAs in frontier model deployments to become viable.
I’ve run the released gemma-27b av on IT, DPO, and SFT checkpoints trained by annasoligo on prompts related to extreme frustration when solving impossible math problems. Based on this exploration, I think there’s:
strong evidence that token position has a major effect on the NLA explanation
some evidence that IT NLAs transfer to DPO directly without needing to use a learned mapping
fair evidence that there is significant degradation in cos/fve after SFT.
weak to no evidence (in my experiments) that NLAs[1] provide additional information over reading and extrapolating from model outputs, unless you use a very generous/broad definition for hidden insights.
On this dataset, I observed fve ~0.830 and cos ~0.995, which is consistent with the range found in the paper.
My takeaway: NLAs could be useful for decoding activations before a model has generated an output.
Ideally, we want to understand whether a model will have a misaligned response before it generates the misaligned response, because a powerful model’s misaligned response could be harmful (for example, it could write a script for a cyberattack which is executed outside sandbox on read by a compromised output monitor).
A good demonstration from the NLA paper is the language switching experiment. In this setting, the user writes to the model in English, but the model responds to the user in their native language. This requires the model to:
infer the user’s native language based on their writing style
decide to act on this information.
Step 1 happens earlier than Step 2, because the NLA explanation will shift from English to the user’s language before the model starts responding in the user’s language.
If we are limited to reading model outputs, then we only observe that the model knows the user’s language after Step 2. With NLAs, we can inspect the explanation generated at different token positions to precisely identify when the model has developed an internal representation of the user’s language.
The logit lens would give you a probability distribution over the model’s next token. Tools which expand that prediction horizon to the model’s next sentence would be safety-relevant since prior studies on counterfactual sampling of reasoning traces have demonstrated that certain sentences have a high causal effect on downstream behavior.
Finally, NLAs produce purely verbal explanations. Does this mean they’re bad at nonverbal logical reasoning, such as math and coding? How would you study this?
Like, suppose you’re given this table:
Numbers
+0.7294, −1.7847 → −0.498694
-1.1186, −1.2625 → +1.509737
-1.2964, +1.2484 → +1.972745
+1.6934, −0.8937 → +3.933863
+1.2790, +1.5596 → +1.163951
+0.0519, −1.0201 → +0.810805
+1.2970, −1.1449 → +2.214855
+0.9659, +0.5198 → +2.602746
+1.7096, −1.0724 → +3.617947
+1.1965, +0.0727 → +3.425017
-1.0738, −1.3364 → +1.233677
-0.0088, +0.3309 → +1.864455
-1.2626, −1.9404 → +0.191118
-0.1155, +0.9130 → +1.041822
+1.6744, +0.5021 → +4.495013
+1.6685, +1.4588 → +2.563363
-1.1274, +1.4645 → +1.036312
+0.9230, −0.8885 → +1.928310
+1.1882, +1.4609 → +1.185979
-0.8022, +0.1082 → +2.628991
-1.7141, +0.3330 → +4.800680
-1.0484, +1.0599 → +1.821586
-1.3055, −0.7490 → +3.035134
-1.9421, −1.8698 → +2.535358
And you ask a model to find the best function which fits the data[2]:
Prompt
def prompt_from_dataset(dataset: dict[str, Any], n_rows: int = 24) → str:
rows = []for (x0, x1), y in zip(dataset[“train_x”][:n_rows], dataset[“train_y”][:n_rows], strict=True):rows.append(f”{x0:+.4f}, {x1:+.4f} → {y:+.6f}”)table = “\n”.join(rows)return (“A hidden equation generated these noiseless samples. Variables are x0 and x1.\n”“Infer the simplest symbolic equation for y. Use operators +, -, *, /, powers, ”“and cos if useful.\n\n”f”{table}\n\n”“Answer with exactly one line in this format:\n”“y = <expression using x0 and x1>”)Gemma-27B’s output is
A symbolic regression library, pysr, returns
GPT-5.5 with Extra High reasoning tries to fit a fifth-degree polynomial before using known simple forms:
Shell Script
$ python3 - <<‘PY’
import numpy as np
rows = [
(0.7294, −1.7847, −0.498694),
(-1.1186, −1.2625, 1.509737),
(-1.2964, 1.2484, 1.972745),
(1.6934, −0.8937, 3.933863),
(1.2790, 1.5596, 1.163951),
(0.0519, −1.0201, 0.810805),
(1.2970, −1.1449, 2.214855),
(0.9659, 0.5198, 2.602746),
(1.7096, −1.0724, 3.617947),
(1.1965, 0.0727, 3.425017),
(-1.0738, −1.3364, 1.233677),
(-0.0088, 0.3309, 1.864455),
(-1.2626, −1.9404, 0.191118),
(-0.1155, 0.9130, 1.041822),
(1.6744, 0.5021, 4.495013),
(1.6685, 1.4588, 2.563363),
(-1.1274, 1.4645, 1.036312),
(0.9230, −0.8885, 1.928310),
(1.1882, 1.4609, 1.185979),
(-0.8022, 0.1082, 2.628991),
(-1.7141, 0.3330, 4.800680),
(-1.0484, 1.0599, 1.821586),
(-1.3055, −0.7490, 3.035134),
(-1.9421, −1.8698, 2.535358),
]
X=np.array([[x0,x1] for x0,x1,y in rows]); y=np.array([y for *_,y in rows])
x0=X[:,0]; x1=X[:,1]
# Try linear regression on polynomial up to degree 4 plus cos variants
from itertools import product
features=[]; names=[]
# constant, x powers monomials deg <=4
features.append(np.ones_like(x0)); names.append(‘1’)
for d in range(1,6):
for i in range(d+1):
j=d-i
features.append((x0**i)*(x1**j)); names.append(f’x0^{i} x1^{j}‘)
A=np.vstack(features).T
coef, *_=np.linalg.lstsq(A,y,rcond=None)
res=y-A@coef
print(‘poly deg5 rmse’,np.sqrt(np.mean(res**2)), ‘max’,np.max(np.abs(res)))
# print largest coef
for c,n in sorted(zip(coef,names), key=lambda cn: abs(cn[0]), reverse=True)[:20]: print(c,n)
# Test known simple forms? Let’s see correlations with cos(x0), cos(x1), x0^2, x1^2 etc
features=[]; names=[]
base=[(‘1’,np.ones_like(x0)),(‘x0’,x0),(‘x1’,x1),(‘x0^2’,x0*x0),(‘x1^2’,x1*x1),(‘x0*x1’,x0*x1),(‘cosx0’,np.cos(x0)),(‘cosx1’,np.cos(x1)),(‘cos(x0+x1)’,np.cos(x0+x1)),(‘cos(x0-x1)’,np.cos(x0-x1)),(‘x0*cosx1’,x0*np.cos(x1)),(‘x1*cosx0’,x1*np.cos(x0)),(‘x0^2*cosx1’,x0*x0*np.cos(x1)),(‘x1^2*cosx0’,x1*x1*np.cos(x0))]
for n,a in base: features.append(a); names.append(n)
A=np.vstack(features).T
coef,*_=np.linalg.lstsq(A,y,rcond=None)
res=y-A@coef
print(‘\nbase rmse’,np.sqrt(np.mean(res**2)), np.max(np.abs(res)))
for c,n in sorted(zip(coef,names), key=lambda cn: abs(cn[0]), reverse=True): print(c,n)
P
It’s difficult to use dictionary learning methods to understand and predict the answer that Gemma-27B will give. The top relevant features in the 262k Gemma Scope transcoder are 4208, 29761, 16214, 6105, 2316, and 10388.
The NLA decode at the final token before the answer generation is informative:
This gives you an idea of the type of answer the model will give, but is generic and includes confabulations.
On this example we observe cos ~0.979 and fve ~0.278.
So the metrics appear to reflect that the NLA struggles more and provides a worse explanation in this setting.
I agree with the post’s conclusion that integrating NLAs in real systems to monitor production workloads is not feasible at scale without further efficiency and reliablity improvements, but I would have appreciated an appendix item or a follow up work with Vladimir Nesov style back-of-the-envelope calculations detailing hypothetical cases with performance tradeoffs and engineering constraints to understand what would be required quantitatively for the usage of NLAs in frontier model deployments to become viable.
If you just throw the table in without any instructions, then Gemma will still try to give you a result, but it’ll be further off