emiddell

• emiddell 15 Dec 2023 10:06 UTC

  4 points

  1

  on: Map­ping the se­man­tic void: Strange go­ings-on in GPT em­bed­ding spaces

  In higher dimensions most of the volume of a n-sphere is found close to its shell.

  The volume of such a sphere is $V_n\left(R\right)\propto R^n$. [1]

  The ratio of the volume of a shell to the rest of the ball is

  $\frac{V_n(R+\Delta )-V_n\left(R\right)}{V_n\left(R\right)}=(1+\frac{\Delta }{R}{)}^n-1$

  which grows quickly with n.

  The embedding places the tokens in areas where there is the most space to accommodate them.

  [1] https://​​en.wikipedia.org/​​wiki/​​Volume_of_an_n-ball