Aryan Bhatt comments on Brief Notes on Transformers

Aryan Bhatt 22 Nov 2022 7:36 UTC
2 points
0
AF
$W_{Q}^{T} W_{K} / d_{k}$
Sorry for the pedantic comment, but I think you might’ve meant to have $\sqrt{d_{k}}$ in the denominator here.
- Adam Jermyn 23 Nov 2022 2:40 UTC
  LW: 2 AF: 1
  0
  AF Parent
  Ah that’s right. Will edit to fix.