Adam Pearce

• Adam Pearce 16 Aug 2023 2:50 UTC

  1 point

  0

  in reply to: salman.mohammadi’s comment on: An in­ter­ac­tive in­tro­duc­tion to grokking and mechanis­tic interpretability

  Lots of custom d3 https://​​github.com/​​PAIR-code/​​ai-explorables/​​tree/​​master/​​source/​​grokking

• Adam Pearce 7 Aug 2023 21:28 UTC

  7 points

  2

  on: Grow­ing Bon­sai Net­works with RNNs

  • The optimization section of Learning Transformer Programs might work with your task/​model

  • You’ve probably seen David Ha’s work, but something like https://​​es-clip.github.io/​​ could be a good starting point for dropping backprop.

  • The exotic activation function almost feels like cheating? Like I want the model the model to discover these useful structures, then try to understand them. But trying to do everything at once may be too hard.

  • Incredibility minor, but changing from onchange to oninput and dropping the animation will make the slider feel much slicker.

An in­ter­ac­tive in­tro­duc­tion to grokking and mechanis­tic interpretability