Algon comments on Paper: Transformers learn in-context by gradient descent

[Algon]

If true, it feels like stories of how models with attention learn to be deceptive are simpler than I thought they were.

EDIT: A somewhat enlightening twitter thread by the authors.