I think that in reality there is some deep generalization happening, but by default “neural networks are lazy”, so majority of out-distribution work is done by enormous amount of shallow circuits.
Not sure what you mean by “deep generalization”, but in general, I don’t see how generalization is incompatible with shallow circuits. I haven’t read the Tegmark paper you linked, but if it’s something like Neel Nanda’s grokking of modular arithmetic work, that circuit was also pretty shallow (one-layer transformer IIRC).
I think it’s explainable by the fact that modular arithmetic is not very complicated. By deep generalization I mean like “semantically rich world model encoded in relatively small number of circuits”. You can have world model encoded in large number of shallow circuits, but I think that my point about resulting in-distribution-only alignment probably stands.
Not sure what you mean by “deep generalization”, but in general, I don’t see how generalization is incompatible with shallow circuits. I haven’t read the Tegmark paper you linked, but if it’s something like Neel Nanda’s grokking of modular arithmetic work, that circuit was also pretty shallow (one-layer transformer IIRC).
I think it’s explainable by the fact that modular arithmetic is not very complicated. By deep generalization I mean like “semantically rich world model encoded in relatively small number of circuits”. You can have world model encoded in large number of shallow circuits, but I think that my point about resulting in-distribution-only alignment probably stands.