SGD finds algorithms. Before the DL revolution, science studied such algorithms. Now, the algorithms become inference without as much as a second glance. With sufficient abundance of general intelligence brought about by AGI, interpretability might get a lot out of studying the circuits SGD discovers. Once understood, the algorithms could be put to more efficient use, instead of remaining implicit in neural nets and used for thinking together with all the noise that remains from the search.
SGD finds algorithms. Before the DL revolution, science studied such algorithms. Now, the algorithms become inference without as much as a second glance. With sufficient abundance of general intelligence brought about by AGI, interpretability might get a lot out of studying the circuits SGD discovers. Once understood, the algorithms could be put to more efficient use, instead of remaining implicit in neural nets and used for thinking together with all the noise that remains from the search.