Apologies if it’s obvious, but why the focus on SGD? I’m assuming it’s not meant as shorthand for other types of optimization algorithms given the emphasis on SGD’s specific inductive bias, and the Deep Double Descent paper mentions that the phenomena hold across most natural choices in optimizers.
Apologies if it’s obvious, but why the focus on SGD? I’m assuming it’s not meant as shorthand for other types of optimization algorithms given the emphasis on SGD’s specific inductive bias, and the Deep Double Descent paper mentions that the phenomena hold across most natural choices in optimizers.
SGD is meant as a shorthand that includes other similar optimizers like Adam.