I was poking around at a chessformer which mimics human play and made this fun companion app to visualize a lot of the internals of the engine. Check out how attention heads look, how the residual stream evolves, and play around with it if interested!
Watch a chess transformer think
Link post
I was poking around at a chessformer which mimics human play and made this fun companion app to visualize a lot of the internals of the engine. Check out how attention heads look, how the residual stream evolves, and play around with it if interested!