FWIW, one of the authors of the stochastic parrots paper writes
To briefly explain the technology behind the metaphor, LLM training relies on stochasticity — in the form of stochastic gradient descent — to build statistical representations of text-based language. “Parroting” is central to how LLMs learn: Given a bunch of text, the model follows each piece in sequence, and is rewarded for correctly predicting the next piece [1]. The result is a model where each piece of text is associated with numeric information about the sequences it tends to occur in.
This means that LLMs are not designed for verbatim playback of text [2]. Instead, when prompted to generate, they draw on their learned representations to parrot smaller spans of text based on whether they’re a probable continuation of what came before — that’s yet another form of stochasticity. The end result is text that looks human-written because that’s exactly what LLMs have been exposed to.
FWIW, one of the authors of the stochastic parrots paper writes
(tbc I haven’t read most of her post)