abhatt349 comments on An exploration of GPT-2′s embedding weights

abhatt349 8 May 2023 23:31 UTC
2 points
0
Very small typo: when you define LayerNorm, you say $y_{i} = \sum_{j} ({Id}_{i j} - 1_{i} 1_{j}) x_{j}$ when I think you mean $y_{i} = \sum_{j} ({Id}_{i j} - \frac{1_{i} 1_{j}}{768}) x_{j}$ ? Please feel free to ignore if this is wrong!!!