Small foundational puzzle for causal theories of mechanistic interpretability

[Frederik Hytting Jørgensen]

In this post I want to highlight a small puzzle for causal theories of mechanistic interpretability. It purports to show that causal abstractions do not generally correctly capture the mechanistic nature of models.

Consider the following causal model $M$:

Assume for the sake of argument that we only consider two possible inputs: $(0,0)$ and $(1,1)$, that is, $X_1$ and $X_2$ are always equal.^[1]

In this model, it is intuitively clear that $X_1$ is what causes the output $X_5$, and $X_2$ is irrelevant. I will argue that this obvious asymmetry between $X_1$ and $X_2$ is not borne out by the causal theory of mechanistic interpretability.

Consider the following causal model $M^{\ast }$:

Is $M^{\ast }$ a valid causal abstraction of the computation that goes on in $M$? That seems to depend on whether $Y_1$ corresponds to $X_1$ or to $X_2$. If $Y_1$ corresponds to $X_1$, then it seems that $M^{\ast }$ is a faithful representation of $M$. If $Y_1$ corresponds to $X_2$, then $M^{\ast }$ is not intuitively a faithful representation of $M$. Indeed, if $Y_1$ corresponds to $X_2$, then we would get the false impression that $X_2$ is what causes the output $X_5$.

Let’s consider the situation where $Y_1$ corresponds to $X_2$. Specifically, define a mapping between values in the two models

$$\begin{array}{c}\tau (x_1,x_2,x_3,x_4,x_5)=\left({\tau }_1\right(x_1,x_2),{\tau }_2(x_3,x_4),{\tau }_3(x_5\left)\right)\end{array}$$

with

$$\begin{array}{c}{\tau }_1(x_1,x_2)=x_2\\ {\tau }_2(x_3,x_4)=x_3\\ {\tau }_3\left(x_5\right)=x_5,\end{array}$$

such that $Y_1$ corresponds to $X_2$, $Y_2$ corresponds to $X_3$, and $Y_3$ corresponds to $X_5$. How do we define whether $M^{\ast }$ abstracts $M$ under $\tau$? The essential idea is that for every single-node hard intervention $i$ on intermediary nodes in the high-level model $M^{\ast }$, there should be an implementation $\text{Imp}\left(i\right)$ of this intervention on the low-level model such that ^[2]

$$\forall (x_1,x_2)\in \left\{\right(0,0),(1,1\left)\right\}:\tau \left(M_{\text{Imp}\left(i\right)}\right(x_1,x_2\left)\right)=M_i^{\ast }\left({\tau }_1\right(x_1,x_2\left)\right)).$$

Let us be explicit about the implementations of interventions:

$$\begin{array}{c}\text{Imp}\left(\text{do}\right(\text{nothing}\left)\right)=\text{do}\left(\text{nothing}\right)\\ \text{Imp}\left(\text{do}\right(Y_2:=y\left)\right)=\text{do}(X_3:=y),\end{array}$$

for $y\in \{0,1\}$. Now, we can check that the abstraction relationship holds. For example:

$$\begin{array}{c}\tau \left(M_{\text{Imp}\left(\text{do}\right(Y_2=1\left)\right)}\right(0,0\left)\right)=\tau (0,0,1,0,1)=(0,1,1)\\ M_{\text{do}(Y_2=1)}^{\ast }\left({\tau }_1\right(0,0\left)\right)=M_{\text{do}(Y_2=1)}^{\ast }\left(0\right)=(0,1,1).\end{array}$$

That $M^{\ast }$ is a valid causal abstraction of $M$ under $\tau$ shows that the notion of causal abstraction, as I have formalized it here, does not correctly capture the computation that happens on the low level (and I claim that the same is true for similar formalizations, for example, Definition 25 in Geiger et al. 2025). Indeed, if the range of $(X_1,X_2)$ is extended such that $(X_1,X_2)\in \{0,1{\}}^2$, we would now make a wrong prediction if we use $M^{\ast }$ to reason about $M$, for example, because $M^{\ast }\left({\tau }_1\right(1,0\left)\right)=(0,0,0)\ne (0,1,1)=\tau \left(M\right(1,0\left)\right)$.

One might object that as long as the range of $(X_1,X_2)$ is $\left\{\right(1,1),(0,0\left)\right\}$, $M^{\ast }$ should indeed be considered a valid abstraction of $M$ under $\tau$. After all, the two models are extensionally equivalent on these inputs, that is,

$$\begin{array}{cc}\tau \left(M\right(1,1\left)\right)=(1,1,1)& =M^{\ast }\left({\tau }_1\right(1,1\left)\right),\\ \tau \left(M\right(0,0\left)\right)=(0,0,0)& =M^{\ast }\left({\tau }_1\right(0,0\left)\right).\end{array}$$

I think this objection misses the mark. The goal of mechanistic interpretability is to understand the intensional properties of algorithms and potentially use this understanding to make predictions about extensional properties. For example, if you examine the mechanisms of a neural network and find that it is simply adding two inputs, you can use this intensional understanding to make a prediction about the output on any new set of inputs. In our example, $M^{\ast }$ and $\tau$ get the intensional properties of $M$wrong, incorrectly suggesting that it is $X_2$ rather than $X_1$ that causes the output. This incorrect intensional understanding leads to a wrong prediction about extensional behavior once the algorithm is evaluated on a new input not in $\left\{\right(0,0),(1,1\left)\right\}$. While I have not argued this here, I believe that this puzzle cannot be easily fixed, and that it points towards a fundamental limitation of the causal abstraction agenda for mechanistic interpretability, insofar as the definitions are meant to provide mechanistic understanding or guarantees about behavior.

Thanks to Atticus Geiger and Thomas Icard for interesting discussions related to this puzzle. Views are my own.

1. ^

   Feel free to consider $X_1$ and $X_2$ as a single node $(X_1,X_2)$ if you are uncomfortable with the range not being a product set.

2. ^

   Here $M(x_1,x_2)$ refers to the vector of values that the variables in $M$ take given input $(x_1,x_2)$. For example, $M(1,1)=(1,1,1,1,1)$.