ARC does expect to be able to “explain” how a chess-playing AI works, though we expect that to be pretty challenging. It avoids some of the issues with an unknown input distribution, but in terms of the mechanistic explanation itself I think it’s going to be just as hard as an LLM.
We think such an explanation should:
Let you predict how often a chess AI wins against another AI from a given position using a mechanistic analysis of the rules of chess and the dynamics of the network, with strictly better computational efficiency than just playing a bunch of games. (Note that predicting who wins between two AIs requires understanding both of those models, for neural networks those explanations will be learned by gradient descent, but for a hand-coded chess algorithm we’ll need to do some hand-coding of an explanation that we conjecture to be not much harder than the design of the original algorithm.)
It should let you answer a range of questions about the AI that should intuitively be easy “if you understood how the model works.” That would include finding certain kinds of adversarial examples, by identifying and then violating assumptions that the model’s performance is predicated on. Perhaps more importantly, if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens and recognize changes that would reduce that probability.
By the same token it might let you extract the rules of chess to the extent that you had a precise question to which “the rules of chess” were the answer, but would definitely not let you explain the rules of chess to a human (since no humans were involved in the definition of the process). And if you had the rules of chess you would then be able to beat sampling in estimating how often the model would try to make an illegal move and would be in a better position to find cases in which it tried to make an illegal move (though again this might require some chess-specific labor, whose output would be simpler than the rules of chess).
I’m curious how exactly the “if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens” would work.
In traditional interability, a human can just read it the goals. With mechanistic explanations, is it that you could show that fiddling with certain parameters or activations would increase the win-rate (without needing to sample it)? Or that if it has other goals you could find an explanation of why it is following those other goals, thus proving it isn’t always aligned to chess?
You would never learn “The model is not trying to play chess well” (the word “trying” will never appear in any outputs from this system). But when you ask “How likely is the model to win this game of chess?”, you’ll get a lower number in games where the model is not trying (because the explanation necessarily captures the mechanisms by which “trying” causes the probability of victory to be higher).
Similarly for takeover attempts, you won’t learn “the model is trying to take over.” But if you ask your estimator “What’s the chance that something would look horribly wrong if I investigated this incident?” the probability will spike when the model starts trying to take over.
(Unless it takes over in such a way that you would not notice even if you investigated. To get around that you could try to say “Well what if I shut down all the AI and did a big investigation at the moment when it would be most damning?” I think that getting answers to this kind of question would be a huge deal and would get us most of what we want / would require solving the core difficulties in alignment. but it’s still far from bulletproof and at some point we need to roll up our sleeves and deal with the philosophical work I mentioned about referring to the causes for patterns in our observations and not just the observations themselves.)
Hmm okay I think I understand! Basically you’re making it more efficient to answer questions that you could ask about a black box, but would be infeasible to answer for a black box. Like for the takeover, you could theoretically answer the investigation question by testing it, but that would be inefficient to do in real time, but with a mechanistic explanation you could do so efficiently.
(Tangentially related: Katago (a game engine for Go) recently switched over to transformers. And like most game engines, it is strongly superhuman (in a narrow domain), but the neural net is fairly small since the neural net is combined with a tree search algorithm. Might be an easier candidate to explain before LLMs!)
I have actually been independently trying to interpret ARC’s agenda for the case of NNUE (which I also expect to be quite difficult in general, as it is effectively a MLP with binary inputs), so this comment is useful clarification.
Some of these ambitious goals seem to be referring to the combined system of chess AI (search + evaluation), and I think search (especially minimax-based search) poses a much different flavor of interpretability compared to interpreting the raw NN based evaluation. Personally, I think the behavior of search alone is still extremely difficult to understand, despite being mechanistically open.
Would you consider an attempt at only interpreting evaluation (NNUE) to be meaningful? Out of the points you mention, only finding adversarial examples might be partly covered, specifically fortress-like positions where search is not able to overcome evaluator bias. If so, are there other evaluator-only questions or directions that you would consider useful tests of ARC’s notion of mechanistic explanation?
From ARC’s perspective we want to explain the interesting properties of NNUE (for us that’s exactly what it means to “interpret” a model). By far the most interesting behavior of NNUE is that it plays good chess, though there are also various closely related properties like “Its output is highly correlated with the winner of the game.”
For any such properties you would need to start by ensuring that you can mechanistically analyze chess games appropriately. E.g. for a given position under random play you should have a mechanistic estimate for the the probability that white wins which is more accurate than sampling. I think that’s a really interesting and probably very tractable problem—I expect a solution to involve tracking how various features evolve over the course of the game, and then to track increasingly high-order interactions between features as you improve your precision.
I think this is roughly what you mean by “the behavior of search alone is still extremely difficult to understand, despite being mechanistically open,” although perhaps I’m saying something slightly more extreme in that I’m saying even the outcome of random play is nontrivial to understand. There isn’t a neural network for which we need an explanation, and indeed we think that the algorithm for mechanistically predicting the outcome of chess games should be very simple (roughly speaking we think it should be no more complex than the rules of chess themselves).
If you did that then it would be very reasonable to look at NNUE and estimate e.g. why its outputs are correlated with the outcome of the game under random play. A good understanding of NNUE should let you do that (or even estimate the gradient of that correlation with respect to the weights). I think that’s a very ambitious project that would be a huge deal for ARC if it could succeed, I’d say maybe ~50% of the way to explaining an LLM.
ARC does expect to be able to “explain” how a chess-playing AI works, though we expect that to be pretty challenging. It avoids some of the issues with an unknown input distribution, but in terms of the mechanistic explanation itself I think it’s going to be just as hard as an LLM.
We think such an explanation should:
Let you predict how often a chess AI wins against another AI from a given position using a mechanistic analysis of the rules of chess and the dynamics of the network, with strictly better computational efficiency than just playing a bunch of games. (Note that predicting who wins between two AIs requires understanding both of those models, for neural networks those explanations will be learned by gradient descent, but for a hand-coded chess algorithm we’ll need to do some hand-coding of an explanation that we conjecture to be not much harder than the design of the original algorithm.)
It should let you answer a range of questions about the AI that should intuitively be easy “if you understood how the model works.” That would include finding certain kinds of adversarial examples, by identifying and then violating assumptions that the model’s performance is predicated on. Perhaps more importantly, if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens and recognize changes that would reduce that probability.
By the same token it might let you extract the rules of chess to the extent that you had a precise question to which “the rules of chess” were the answer, but would definitely not let you explain the rules of chess to a human (since no humans were involved in the definition of the process). And if you had the rules of chess you would then be able to beat sampling in estimating how often the model would try to make an illegal move and would be in a better position to find cases in which it tried to make an illegal move (though again this might require some chess-specific labor, whose output would be simpler than the rules of chess).
Interesting!
I’m curious how exactly the “if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens” would work.
In traditional interability, a human can just read it the goals. With mechanistic explanations, is it that you could show that fiddling with certain parameters or activations would increase the win-rate (without needing to sample it)? Or that if it has other goals you could find an explanation of why it is following those other goals, thus proving it isn’t always aligned to chess?
You would never learn “The model is not trying to play chess well” (the word “trying” will never appear in any outputs from this system). But when you ask “How likely is the model to win this game of chess?”, you’ll get a lower number in games where the model is not trying (because the explanation necessarily captures the mechanisms by which “trying” causes the probability of victory to be higher).
Similarly for takeover attempts, you won’t learn “the model is trying to take over.” But if you ask your estimator “What’s the chance that something would look horribly wrong if I investigated this incident?” the probability will spike when the model starts trying to take over.
(Unless it takes over in such a way that you would not notice even if you investigated. To get around that you could try to say “Well what if I shut down all the AI and did a big investigation at the moment when it would be most damning?” I think that getting answers to this kind of question would be a huge deal and would get us most of what we want / would require solving the core difficulties in alignment. but it’s still far from bulletproof and at some point we need to roll up our sleeves and deal with the philosophical work I mentioned about referring to the causes for patterns in our observations and not just the observations themselves.)
Hmm okay I think I understand! Basically you’re making it more efficient to answer questions that you could ask about a black box, but would be infeasible to answer for a black box. Like for the takeover, you could theoretically answer the investigation question by testing it, but that would be inefficient to do in real time, but with a mechanistic explanation you could do so efficiently.
(Tangentially related: Katago (a game engine for Go) recently switched over to transformers. And like most game engines, it is strongly superhuman (in a narrow domain), but the neural net is fairly small since the neural net is combined with a tree search algorithm. Might be an easier candidate to explain before LLMs!)
I have actually been independently trying to interpret ARC’s agenda for the case of NNUE (which I also expect to be quite difficult in general, as it is effectively a MLP with binary inputs), so this comment is useful clarification.
Some of these ambitious goals seem to be referring to the combined system of chess AI (search + evaluation), and I think search (especially minimax-based search) poses a much different flavor of interpretability compared to interpreting the raw NN based evaluation. Personally, I think the behavior of search alone is still extremely difficult to understand, despite being mechanistically open.
Would you consider an attempt at only interpreting evaluation (NNUE) to be meaningful? Out of the points you mention, only finding adversarial examples might be partly covered, specifically fortress-like positions where search is not able to overcome evaluator bias. If so, are there other evaluator-only questions or directions that you would consider useful tests of ARC’s notion of mechanistic explanation?
From ARC’s perspective we want to explain the interesting properties of NNUE (for us that’s exactly what it means to “interpret” a model). By far the most interesting behavior of NNUE is that it plays good chess, though there are also various closely related properties like “Its output is highly correlated with the winner of the game.”
For any such properties you would need to start by ensuring that you can mechanistically analyze chess games appropriately. E.g. for a given position under random play you should have a mechanistic estimate for the the probability that white wins which is more accurate than sampling. I think that’s a really interesting and probably very tractable problem—I expect a solution to involve tracking how various features evolve over the course of the game, and then to track increasingly high-order interactions between features as you improve your precision.
I think this is roughly what you mean by “the behavior of search alone is still extremely difficult to understand, despite being mechanistically open,” although perhaps I’m saying something slightly more extreme in that I’m saying even the outcome of random play is nontrivial to understand. There isn’t a neural network for which we need an explanation, and indeed we think that the algorithm for mechanistically predicting the outcome of chess games should be very simple (roughly speaking we think it should be no more complex than the rules of chess themselves).
If you did that then it would be very reasonable to look at NNUE and estimate e.g. why its outputs are correlated with the outcome of the game under random play. A good understanding of NNUE should let you do that (or even estimate the gradient of that correlation with respect to the weights). I think that’s a very ambitious project that would be a huge deal for ARC if it could succeed, I’d say maybe ~50% of the way to explaining an LLM.