In principle one can just duplicate layers and then keep training until they differentiate enough to make use of the extra parameters. That works because the residual stream is modified relatively little by each layer i.e. a layer can still understand its own output as input.
There is a whole niche in open source AI of Frankenstein models that are weirdly patched together with duplicated or deleted layers. But that is probably extremely inefficient and limited compared to training from scratch.
In principle one can just duplicate layers and then keep training until they differentiate enough to make use of the extra parameters. That works because the residual stream is modified relatively little by each layer i.e. a layer can still understand its own output as input.
There is a whole niche in open source AI of Frankenstein models that are weirdly patched together with duplicated or deleted layers. But that is probably extremely inefficient and limited compared to training from scratch.