Adam Pearce
Karma: 27
The optimization section of Learning Transformer Programs might work with your task/model
You’ve probably seen David Ha’s work, but something like https://es-clip.github.io/ could be a good starting point for dropping backprop.
The exotic activation function almost feels like cheating? Like I want the model the model to discover these useful structures, then try to understand them. But trying to do everything at once may be too hard.
Incredibility minor, but changing from
onchangetooninputand dropping the animation will make the slider feel much slicker.
Lots of custom d3 https://github.com/PAIR-code/ai-explorables/tree/master/source/grokking