Of course, when I asked Jürgen Schmidhuber why did it took 18 years after they published LSTMs in 1997 till this set of ideas got transferred from the recurrent setting into the deep feedforward setting, he replied that that was indeed a very good question :-) His own lab introduced both LSTMs and Highway Networks, but somehow it took all this time. All these architectures, like vanilla MLPs and CNNs and AlexNet, precede 2015, and residual streams in the feedforward nets is one of those numerous cases when a very simple discovery got inexplicably delayed for decades.
In any case, the transition to ResNets back then was associated with a performance jump, and ResNets rapidly became the de-facto standard back then, since they allowed to train much deeper nets without triggering the phenomenon of “vanishing gradients”. The Transformers simply inherited this feature.
This idea became mainstream in 2015 with the introduction of Highway Networks, https://arxiv.org/abs/1505.00387 and then, more prominently, ResNets, https://arxiv.org/abs/1512.03385 (Deep Residual Learning for Image Recognition, Google Scholar counts more than 300000 citations on this one).
Of course, when I asked Jürgen Schmidhuber why did it took 18 years after they published LSTMs in 1997 till this set of ideas got transferred from the recurrent setting into the deep feedforward setting, he replied that that was indeed a very good question :-) His own lab introduced both LSTMs and Highway Networks, but somehow it took all this time. All these architectures, like vanilla MLPs and CNNs and AlexNet, precede 2015, and residual streams in the feedforward nets is one of those numerous cases when a very simple discovery got inexplicably delayed for decades.
In any case, the transition to ResNets back then was associated with a performance jump, and ResNets rapidly became the de-facto standard back then, since they allowed to train much deeper nets without triggering the phenomenon of “vanishing gradients”. The Transformers simply inherited this feature.