I think the following would be an intermediate case between simple networks and LLMs: chess AIs
In a world where ARC is successful, what does a mechanistic explanation of, say, Leela Chess Zero or Stockfish NNUE look like?
In traditional mechanistic interpretability, the answer is “a chess playing algorithm that isn’t a black box”.
With ARC’s approach, is it just that we could find adversarial inputs faster than sampling? We could eliminate adversarial inputs? Can we somehow “align” the models better to chess based on the explanation?
Since the explanation isn’t human readable, I’m assuming we wouldn’t actually be able to find the rules of chess in the explanation. Like we could find an “explanation” for why it only plays legal moves but that explanation would itself be a giant mathematical artifact. And the explanation for why it wins against non-NN chess AIs would be even more inscrutable!
I think the following would be an intermediate case between simple networks and LLMs: chess AIs
In a world where ARC is successful, what does a mechanistic explanation of, say, Leela Chess Zero or Stockfish NNUE look like?
In traditional mechanistic interpretability, the answer is “a chess playing algorithm that isn’t a black box”.
With ARC’s approach, is it just that we could find adversarial inputs faster than sampling? We could eliminate adversarial inputs? Can we somehow “align” the models better to chess based on the explanation?
Since the explanation isn’t human readable, I’m assuming we wouldn’t actually be able to find the rules of chess in the explanation. Like we could find an “explanation” for why it only plays legal moves but that explanation would itself be a giant mathematical artifact. And the explanation for why it wins against non-NN chess AIs would be even more inscrutable!