I’m surprised you put the emphasis on how Gaussian your curves are, while your curves are much less Gaussian that you would naively expect if you agreed with the “LLM are a bunch of small independent heuristic” argument.
Even ignoring outliers, some of your distributions don’t look like Gaussian distributions to me. In Geogebra, exponential decays fit well, Gaussians don’t.
I think your headlines are misleading, and that you’re providing evidence against “LLM are a bunch of small independent heuristic”.
This is very interesting. The OP doesn’t contain any specific evidence of Gaussianness, so it would be helpful if they could provide an elaboration of what evidence lead them to conclude these are Gaussian.
When investigating distributions from a completely different source, I thought for a while
these are all pretty gaussian! except for the bits on the right, and I know where those come from
then recognized that their tails were wider and thought
maybe they’re … student-t or something? [confused]
and eventually realized
oh, of course, this is a mixture of lots of gaussians; each natural class induces a gaussian in the limit, but this distribution represents a whole slew of natural classes, and most of their means are quite similar, so yeah, it’s gonna look like a fatter-tailed gaussian with some “outliers”
and I think this probably happens quite a lot, because while CLT ==> gaussian, nature rarely says “here’s exactly one type of object!”
If you sum enough Gaussians you can get close to any distribution you want. I’m not sure what the information behind “it’s Gaussian” in this context. (It clearly doesn’t look like a mixture of a few Gaussians...)
It clearly doesn’t look like a mixture of a few Gaussians
It does to me. If their means are close enough compared to their variances, it’ll look like a unimodal distribution. For a good example, a student t distribution is a mixture of gaussians with the same mean and variances differing in a certain way, and it looks exactly like these images.
I’m surprised you put the emphasis on how Gaussian your curves are, while your curves are much less Gaussian that you would naively expect if you agreed with the “LLM are a bunch of small independent heuristic” argument.
Even ignoring outliers, some of your distributions don’t look like Gaussian distributions to me. In Geogebra, exponential decays fit well, Gaussians don’t.
I think your headlines are misleading, and that you’re providing evidence against “LLM are a bunch of small independent heuristic”.
This is very interesting. The OP doesn’t contain any specific evidence of Gaussianness, so it would be helpful if they could provide an elaboration of what evidence lead them to conclude these are Gaussian.
When investigating distributions from a completely different source, I thought for a while
these are all pretty gaussian! except for the bits on the right, and I know where those come from
then recognized that their tails were wider and thought
maybe they’re … student-t or something? [confused]
and eventually realized
oh, of course, this is a mixture of lots of gaussians; each natural class induces a gaussian in the limit, but this distribution represents a whole slew of natural classes, and most of their means are quite similar, so yeah, it’s gonna look like a fatter-tailed gaussian with some “outliers”
and I think this probably happens quite a lot, because while CLT ==> gaussian, nature rarely says “here’s exactly one type of object!”
If you sum enough Gaussians you can get close to any distribution you want. I’m not sure what the information behind “it’s Gaussian” in this context. (It clearly doesn’t look like a mixture of a few Gaussians...)
It does to me. If their means are close enough compared to their variances, it’ll look like a unimodal distribution. For a good example, a student t distribution is a mixture of gaussians with the same mean and variances differing in a certain way, and it looks exactly like these images.
See the first image here: https://en.m.wikipedia.org/wiki/Student’s_t-distribution