I agree that validation is the real variable, but I wonder if that points somewhere sharper than “trust inversely proportional to complexity”. Complexity is a property of the tool, but falsifiability is a property of the claim, and the two can come apart in both directions.
I recently worked on a small character-level transformer, and your points got me reflecting on my methods. Reading the OV circuit directly, I found a high self-copy logit on quotes (”), and a bracket completion signal that looked like parenthesis-matching ( attending to “(” and boosting ”)” ), and concluded I was looking at learned quotation and parenthesis grammar. Both were wrong. What broke the stories was also simple. I threw a few test sentences through and read where the QK attention went (it tracked induction successors and the latest bracket, with no depth tracking). Simple, legible tools handed me two false conclusions, but simple checks caught them. (Admittedly, my checks were still observational. I was reading attention patterns on hand-written prompts. So this could still be wrong under stronger tests.)
So in my case, it wasn’t the complexity of the tool, it was whether the claim was falsifiable. The pragmatic interp pivot makes a similar point: that rigor comes from a scientific mindset and falsification, not from the method.
Which makes me wonder if the real question isn’t “is this tool trustworthy” but “is this specific claim testable”. A simple probe can support an untestable story, and an NLA explanation could generate a verifiable prediction. Testing wouldn’t track complexity one-to-one. Testing could offset the distrust of complexity.
Though I do see the hardest cases (sandbagging, scheming) may not yield testable predictions, and so complexity has to be the proxy, which I take as your parenthetical point. But maybe that suggests where the line gets drawn: some claims can be checked directly (does X ⇒ Y as predicted?) even when the big question (is the model scheming?) can’t be answered directly.
Still finding my footing in this area though, so happy to hear what I’m not considering or misunderstood.
I agree that validation is the real variable, but I wonder if that points somewhere sharper than “trust inversely proportional to complexity”. Complexity is a property of the tool, but falsifiability is a property of the claim, and the two can come apart in both directions.
I recently worked on a small character-level transformer, and your points got me reflecting on my methods. Reading the OV circuit directly, I found a high self-copy logit on quotes (”), and a bracket completion signal that looked like parenthesis-matching ( attending to “(” and boosting ”)” ), and concluded I was looking at learned quotation and parenthesis grammar. Both were wrong. What broke the stories was also simple. I threw a few test sentences through and read where the QK attention went (it tracked induction successors and the latest bracket, with no depth tracking). Simple, legible tools handed me two false conclusions, but simple checks caught them. (Admittedly, my checks were still observational. I was reading attention patterns on hand-written prompts. So this could still be wrong under stronger tests.)
So in my case, it wasn’t the complexity of the tool, it was whether the claim was falsifiable. The pragmatic interp pivot makes a similar point: that rigor comes from a scientific mindset and falsification, not from the method.
Which makes me wonder if the real question isn’t “is this tool trustworthy” but “is this specific claim testable”. A simple probe can support an untestable story, and an NLA explanation could generate a verifiable prediction. Testing wouldn’t track complexity one-to-one. Testing could offset the distrust of complexity.
Though I do see the hardest cases (sandbagging, scheming) may not yield testable predictions, and so complexity has to be the proxy, which I take as your parenthetical point. But maybe that suggests where the line gets drawn: some claims can be checked directly (does X ⇒ Y as predicted?) even when the big question (is the model scheming?) can’t be answered directly.
Still finding my footing in this area though, so happy to hear what I’m not considering or misunderstood.