Good analysis, something like this is why I’m also not as much concerned about superhuman LLMs.
One thing I’d be curious about is how Universal Transformers modify your conclusions. These have variable per-symbol processing length, so in theory they could learn algorithms that a transformer with finite calculation cannot.
On the other hand… UTs aren’t really being used anywhere (so far as I know) so they probably have other big pitfalls. Right now I do I think something like your analysis would probably apply to any realistic transformer modifications, and that UTs aren’t about to destroy transformers, but that’s a hunch and I wish I had better reasoning for why.
Good analysis, something like this is why I’m also not as much concerned about superhuman LLMs.
One thing I’d be curious about is how Universal Transformers modify your conclusions. These have variable per-symbol processing length, so in theory they could learn algorithms that a transformer with finite calculation cannot.
On the other hand… UTs aren’t really being used anywhere (so far as I know) so they probably have other big pitfalls. Right now I do I think something like your analysis would probably apply to any realistic transformer modifications, and that UTs aren’t about to destroy transformers, but that’s a hunch and I wish I had better reasoning for why.