Apologies if it’s obvious, but why the focus on SGD? I’m assuming it’s not meant as shorthand for other types of optimization algorithms given the emphasis on SGD’s specific inductive bias, and the Deep Double Descent paper mentions that the phenomena hold across most natural choices in optimizers.
Apologies if it’s obvious, but why the focus on SGD? I’m assuming it’s not meant as shorthand for other types of optimization algorithms given the emphasis on SGD’s specific inductive bias, and the Deep Double Descent paper mentions that the phenomena hold across most natural choices in optimizers.