Why do AIs write using the “AI style”? (Overuse of words like “delve” or constructions like “it is not X, it is Y”.) I mean, they are trained on human texts, people call them “stochastic parrots”, so how is it possible for them to develop a distinct style? Isn’t that like the only thing they—practically by definition—shouldn’t do? I would expect them to write text that superficially seems like written by humans, but if you look closely, you see that the style is there but the substance is not. Instead, the style is wrong.
There are many things that I don’t understand about LLMs (practically all of them), but this one bothers me because I would expect the exact opposite.
(Is it possible that the “AI style” is actually a style of some category of humans that produce a lot of texts that for some mysterious reason most people never see, so those texts are over-represented in the learning data? Journalists maybe?)
There is an important distinction to be made between base models, which are next token predictors (and write in a very human like manner) and the chatbots people normally use, despite both being called “LLMs”. Those involve taking a base model and then subjecting it to further instruction tuning and Reinforcement Learning from Human Feedback.
My understanding is that a lot of the stylistic quirks of the latter arise from the fact that OpenAI (and other companies) either personally or through outsourcing hired Third World contractors. They were cheap and reasonably fluent in English. However, a large number were Nigerian, and this resulted in a model slightly biased towards Nigerian English. They’re far more fond of “delve” than Western English speakers, but most people wouldn’t know that, because how often were they reading Nigerian literature or media? It’s particularly rampant in their business or corporate comms.
Then you have other issues stemming from RLHF. Certain aspects of the general chatbot persona were overly engrained, as preference feedback biased towards verbosity and enthusiasm. And now that the output of ChatGPT 3.5 and onwards is all over the wider internet, to an extent, new base models expect that that’s how a chatbot talks. This likely bleeds into the chatbots that are built on the contaminated base model, since they have very strong priors that they’re a chatbot in the first place.
That doesn’t mean that there isn’t significant variance in how current models speak. Claude has a rather distinct personality and innate voice, even within the various GPT models, o3 was highly terse and fond of dense jargon. With the availability of custom instructions, you can easily get them to talk you in just about any style you prefer.
A much simpler Markov chain, trained apparently on “the King James Bible and the Structure and Interpretation of Computer Programs” (I think I read that “The Art of Unix Programming” or something is also in there). Examples:
then shall they call upon me, but I will not cause any information to be accumulated on the stack.
How much more are ye better than the ordered-list representation
evaluating the operator might modify env, which will be the hope of unjust men
If you imagine that style 1 has traits A1, B1, and C1, and style 2 has traits A2, B2, and C2, then you could end up with style 3 having traits A1, B2, and C1, which is a novel combination. Depending on your criteria of “style”, this might count as a new style. Here it’s pretty clumsy and heavy-handed, and it looks more like switching between style 1 and style 2 (IIRC this particular Markov chain uses the last 3 or 4 words, which is higher than usual and more likely to just reproduce chains of text from the original); but if you imagine there being 100 styles, each having 1000 traits, it seems much more likely that the resulting thing would qualify as a “new style” by a layman’s judgment.
One possible contributor: posttraining involves chat transcripts in the desired style (often, nowadays, generated by an older LLM), and I suspect that in learning to imitate the format models also learn to imitate the tone (and to overfit, at that; perhaps it’s due to having only a few examples relative to the size of the corpus, but this is merely idle speculation). (The consensus on twitter seemed to be that “delve” in particular was a consequence of human writing; it’s used far more commonly in African English than in American, and OpenAI outsourced data labeling to save on costs.) I haven’t noticed nearly as much of a consistent flavor in my limited experimentation with base models, so I think posttraining must make it worse even if it’s not the cause.
I said “delve” was overused by ChatGPT compared to the internet at large. But there’s one part of the internet where “delve” is a much more common word: the African web. In Nigeria, “delve” is much more frequently used in business English than it is in England or the US. So the workers training their systems provided examples of input and output that used the same language, eventually ending up with an AI system that writes slightly like an African.
Out of my ass, the AI style is an artifact of the details of subtle mistakes in ChatGPT 3.5’s post training, propagated by enormous quantities of intentional and unintentional dataset pollution.
Why do AIs write using the “AI style”? (Overuse of words like “delve” or constructions like “it is not X, it is Y”.) I mean, they are trained on human texts, people call them “stochastic parrots”, so how is it possible for them to develop a distinct style? Isn’t that like the only thing they—practically by definition—shouldn’t do? I would expect them to write text that superficially seems like written by humans, but if you look closely, you see that the style is there but the substance is not. Instead, the style is wrong.
There are many things that I don’t understand about LLMs (practically all of them), but this one bothers me because I would expect the exact opposite.
(Is it possible that the “AI style” is actually a style of some category of humans that produce a lot of texts that for some mysterious reason most people never see, so those texts are over-represented in the learning data? Journalists maybe?)
There is an important distinction to be made between base models, which are next token predictors (and write in a very human like manner) and the chatbots people normally use, despite both being called “LLMs”. Those involve taking a base model and then subjecting it to further instruction tuning and Reinforcement Learning from Human Feedback.
My understanding is that a lot of the stylistic quirks of the latter arise from the fact that OpenAI (and other companies) either personally or through outsourcing hired Third World contractors. They were cheap and reasonably fluent in English. However, a large number were Nigerian, and this resulted in a model slightly biased towards Nigerian English. They’re far more fond of “delve” than Western English speakers, but most people wouldn’t know that, because how often were they reading Nigerian literature or media? It’s particularly rampant in their business or corporate comms.
Then you have other issues stemming from RLHF. Certain aspects of the general chatbot persona were overly engrained, as preference feedback biased towards verbosity and enthusiasm. And now that the output of ChatGPT 3.5 and onwards is all over the wider internet, to an extent, new base models expect that that’s how a chatbot talks. This likely bleeds into the chatbots that are built on the contaminated base model, since they have very strong priors that they’re a chatbot in the first place.
That doesn’t mean that there isn’t significant variance in how current models speak. Claude has a rather distinct personality and innate voice, even within the various GPT models, o3 was highly terse and fond of dense jargon. With the availability of custom instructions, you can easily get them to talk you in just about any style you prefer.
Consult King James Programming: https://www.tumblr.com/kingjamesprogramming
A much simpler Markov chain, trained apparently on “the King James Bible and the Structure and Interpretation of Computer Programs” (I think I read that “The Art of Unix Programming” or something is also in there). Examples:
If you imagine that style 1 has traits A1, B1, and C1, and style 2 has traits A2, B2, and C2, then you could end up with style 3 having traits A1, B2, and C1, which is a novel combination. Depending on your criteria of “style”, this might count as a new style. Here it’s pretty clumsy and heavy-handed, and it looks more like switching between style 1 and style 2 (IIRC this particular Markov chain uses the last 3 or 4 words, which is higher than usual and more likely to just reproduce chains of text from the original); but if you imagine there being 100 styles, each having 1000 traits, it seems much more likely that the resulting thing would qualify as a “new style” by a layman’s judgment.
One possible contributor: posttraining involves chat transcripts in the desired style (often, nowadays, generated by an older LLM), and I suspect that in learning to imitate the format models also learn to imitate the tone (and to overfit, at that; perhaps it’s due to having only a few examples relative to the size of the corpus, but this is merely idle speculation). (The consensus on twitter seemed to be that “delve” in particular was a consequence of human writing; it’s used far more commonly in African English than in American, and OpenAI outsourced data labeling to save on costs.) I haven’t noticed nearly as much of a consistent flavor in my limited experimentation with base models, so I think posttraining must make it worse even if it’s not the cause.
This Guardian article has a pretty plausible hypothesis imo
Out of my ass, the AI style is an artifact of the details of subtle mistakes in ChatGPT 3.5’s post training, propagated by enormous quantities of intentional and unintentional dataset pollution.