Tom Lieberum comments on Hypothesis: gradient descent prefers general circuits

[Tom Lieberum]

I tried increasing weight decay and increased batch sizes but so far no real success compared to 5x lr. Not going to investigate this further atm.