Lucius Bushnaq comments on [Paper] Stochastic Parameter Decomposition

[Lucius Bushnaq]

I’d have to think about the exact setup here to make sure there’s no weird caveats, but my first thought is that for $W_{\text{in}}$, this ought to be one component per bigram, firing exclusively for that bigram.