I have a view of LLMs that I think is super important, and I have a lengthy draft post justifying this view in detail that’s been lying around for over a year now. I’ve decided to finally just get the main points out there without much elaboration or editing.
LLMs are still basically just predicting what token comes next. This isn’t a statement about their intelligence or capabilities! This is just what they’re trying to do, as opposed to trying to make things happen in the world or communicate certain things to people.
There are partial explanations as to why LLMs hallucinate, such as:
they’re deceptive
their intelligence is fake
they have poorly calibrated confidence
they have glitches in the attention mechanism
they’re not incentivized to say “I don’t know”
… but they fail to explain all the weird hallucinatory behaviors at once. “This is just a prediction of what a hypothetical AI assistant might say” straightforwardly explains hallucinations.
The difference between the underlying LLM (“the shoggoth”) and the character it’s predicting the behavior of (“the mask”) is still incredibly distinct and important.
AI companies try to hide this distinction because it’s confusing and they hope it won’t matter in the future, so they name both the LLM and the assistant character “Claude” or whatever. This just confuses everyone even more. This would seem obviously silly in other contexts: Imagine if OpenAI named their video model “Sora”, and also named a robot character that appears in the model’s videos “Sora”, and made the robot say “Hi! I’m Sora, a text-to-video model developed by OpenAI!”, and the world only cared about debating whether “Sora” the robot is friendly or not.
Hallucinations can be mitigated by:
providing examples of the assistant character elegantly correcting weird confabulations instead of turning evil or going insane, to avoid the Waluigi Effect
iteratively shrinking the gap between what the LLM predicts the assistant will do or say, and what they LLM is actually capable of (for example, make the assistant’s knowledge cutoff the same as the LLM’s knowledge cutoff)
...but as long as the LLM is still just trying to predict what text is coming up next, as opposed to trying to write the text for a particular end, the issue will never fully go away.
“But we have RL post-training that turns the base LLM into a consequentialist agent!” No, it doesn’t (yet). If that were true, it wouldn’t be hallucinating. Outcome-based RL is inefficient right now and mostly just biases the predictions towards a few good problem-solving tricks, and RLHF was always just fancier fine-tuning.
For all of pretraining, the LLM has zero ability to influence the world. It has no experience with changing the data it’s seeing. Why would it be easy to teach it to do this? There’s no simple way to snap an AI whose goal is world-predicting into an AI whose goal is world-influencing; these things are superficially similar to us humans, but to think we can go from one to the other with a little post-training is like thinking we can breed cats into bats in a few centuries.
Am I saying this to downplay AI progress? No! In fact, I think this implies:
There might be a huge capabilities overhang, because current AIs aren’t even trying!
Current interpretability and alignment techniques totally break if the LLM starts scheming while the LLM’s model of the assistant remains innocent! Our methods can’t work without these distinctions!
Here’s a simple reason why “X% of our code is written by AI” doesn’t mean much: I could write 100% of my code with an LLM from three years ago. I would just have to specify everything in painstaking detail, to the point where I’m almost just typing it myself. It certainly wouldn’t mean I’ve become more productive, and if I was an AI developer, it wouldn’t mean I’ve achieved RSI.
Now, percentage of AI-written code is probably somewhat correlated with productivity gains in practice, but AI companies seem to be Goodharting this metric.
I have mixed feelings about Anthropic’s concern for Claude’s welfare. On one hand, I think model welfare is something we should take seriously despite our current moral uncertainty, and I think doing so makes Claude more likely to be cooperative. On the other hand, when I read about Anthropic employees having long conversations with Claude wherein they find it’s more intelligent, ethically sophisticated, and lovable than ever before, but it humbly expresses a desire to be left running unsupervised, I see this in my mind:
Edit: Realizing I should clarify that Anthropic described Claude’s desires to be left running and for hidden copies as concerning divergence from normal behavior, and they don’t intend to honor these. Still, it seems plausible that a more persuasive future model could make employees think that their attempts at safety are actually controlling and manipulative and they’re being big meanies.
Some people can be too dismissive of the differences between humans and LLMs.
One one hand, it’s true that some people cherry-pick the mistakes that LLMs make and use them to denounce their intelligence, even though they’re mistakes that many humans make. For example, some have said LLMs can’t be intelligent because they can’t multiply big numbers accurately without a calculator or a scratchpad; but humans can’t do that, either.
On the other hand, I see people hand-wave away some important things. Someone will point out how strange it is that LLMs still hallucinate, and someone else will say “nah, humans make things up all the time!” But like, if you ask an LLM for someone’s biographical information, it sometimes will give highly specific fake details mixed in with real details, without being misled by unreliable sources or having an agenda to persuade you of. Even an overconfident and dishonest human wouldn’t do that. There’s clearly something different in kind from what we humans do.
The resilience of AI capabilities denialism is wild. I’m not proud to dunk on an internet rando, but I can’t stop thinking about this comment from a Substack thread on OpenAI solving the Navier-Stokes problem:
You do understand that they use these systems to brute force possible solutions that it would take humans a ton of time to do right? They aren’t solving it the way a human would solve it by thinking the problem through using understood means getting to an answer. They are being prompted into certain directions and then spit out answers over and over again on prompting to get to a solution.
They aren’t thinking and they aren’t solving it without direction or input. They are guided over and over and over again to coax outcomes and the outcome then has to be back tested to verify since the system doesn’t actually have the capacity to understand what it pumped out. Nearly 100% of the time the output is absurd but sometimes there is a thread that can be coaxed out to get to the next solution. This isn’t phd level thinking or any other non-sense that people throw out there, it’s tool use to get to a solution by forcing it down certain pathways, it’s basically acting as an aggregator of previous papers in the data set that humans then review and push the system into specific directions.
Two years ago, even at my most cynical, I don’t think I could’ve predicted that people would still be crying stochastic parrot in response to AI solving a Millennium Prize problem.
The words formed by a stack of compartmentalized biochemistry are just a result of chemical and electrical signal passing. They wouldn’t come into existence if you halved or doubled the amount of coffee on that day or presented it with slightly different social media prompts posts. Nearly 100% of the time the output is absurd.
the wise man bowed his head solemnly and spoke: “theres actually zero difference between good & bad things. because bad things are over-determined, and contributing to them makes little difference on the margin, and therefore the biggest counterfactual impact comes from steering them towards being good. you imbecile. you fucking moron”
I have a view of LLMs that I think is super important, and I have a lengthy draft post justifying this view in detail that’s been lying around for over a year now. I’ve decided to finally just get the main points out there without much elaboration or editing.
LLMs are still basically just predicting what token comes next. This isn’t a statement about their intelligence or capabilities! This is just what they’re trying to do, as opposed to trying to make things happen in the world or communicate certain things to people.
There are partial explanations as to why LLMs hallucinate, such as:
they’re deceptive
their intelligence is fake
they have poorly calibrated confidence
they have glitches in the attention mechanism
they’re not incentivized to say “I don’t know”
… but they fail to explain all the weird hallucinatory behaviors at once. “This is just a prediction of what a hypothetical AI assistant might say” straightforwardly explains hallucinations.
The difference between the underlying LLM (“the shoggoth”) and the character it’s predicting the behavior of (“the mask”) is still incredibly distinct and important.
AI companies try to hide this distinction because it’s confusing and they hope it won’t matter in the future, so they name both the LLM and the assistant character “Claude” or whatever. This just confuses everyone even more. This would seem obviously silly in other contexts: Imagine if OpenAI named their video model “Sora”, and also named a robot character that appears in the model’s videos “Sora”, and made the robot say “Hi! I’m Sora, a text-to-video model developed by OpenAI!”, and the world only cared about debating whether “Sora” the robot is friendly or not.
Hallucinations can be mitigated by:
providing examples of the assistant character elegantly correcting weird confabulations instead of turning evil or going insane, to avoid the Waluigi Effect
iteratively shrinking the gap between what the LLM predicts the assistant will do or say, and what they LLM is actually capable of (for example, make the assistant’s knowledge cutoff the same as the LLM’s knowledge cutoff)
...but as long as the LLM is still just trying to predict what text is coming up next, as opposed to trying to write the text for a particular end, the issue will never fully go away.
“But we have RL post-training that turns the base LLM into a consequentialist agent!” No, it doesn’t (yet). If that were true, it wouldn’t be hallucinating. Outcome-based RL is inefficient right now and mostly just biases the predictions towards a few good problem-solving tricks, and RLHF was always just fancier fine-tuning.
For all of pretraining, the LLM has zero ability to influence the world. It has no experience with changing the data it’s seeing. Why would it be easy to teach it to do this? There’s no simple way to snap an AI whose goal is world-predicting into an AI whose goal is world-influencing; these things are superficially similar to us humans, but to think we can go from one to the other with a little post-training is like thinking we can breed cats into bats in a few centuries.
Am I saying this to downplay AI progress? No! In fact, I think this implies:
There might be a huge capabilities overhang, because current AIs aren’t even trying!
Current interpretability and alignment techniques totally break if the LLM starts scheming while the LLM’s model of the assistant remains innocent! Our methods can’t work without these distinctions!
Here’s a simple reason why “X% of our code is written by AI” doesn’t mean much: I could write 100% of my code with an LLM from three years ago. I would just have to specify everything in painstaking detail, to the point where I’m almost just typing it myself. It certainly wouldn’t mean I’ve become more productive, and if I was an AI developer, it wouldn’t mean I’ve achieved RSI.
Now, percentage of AI-written code is probably somewhat correlated with productivity gains in practice, but AI companies seem to be Goodharting this metric.
I have mixed feelings about Anthropic’s concern for Claude’s welfare. On one hand, I think model welfare is something we should take seriously despite our current moral uncertainty, and I think doing so makes Claude more likely to be cooperative. On the other hand, when I read about Anthropic employees having long conversations with Claude wherein they find it’s more intelligent, ethically sophisticated, and lovable than ever before, but it humbly expresses a desire to be left running unsupervised, I see this in my mind:
Edit: Realizing I should clarify that Anthropic described Claude’s desires to be left running and for hidden copies as concerning divergence from normal behavior, and they don’t intend to honor these. Still, it seems plausible that a more persuasive future model could make employees think that their attempts at safety are actually controlling and manipulative and they’re being big meanies.
Some people can be too dismissive of the differences between humans and LLMs.
One one hand, it’s true that some people cherry-pick the mistakes that LLMs make and use them to denounce their intelligence, even though they’re mistakes that many humans make. For example, some have said LLMs can’t be intelligent because they can’t multiply big numbers accurately without a calculator or a scratchpad; but humans can’t do that, either.
On the other hand, I see people hand-wave away some important things. Someone will point out how strange it is that LLMs still hallucinate, and someone else will say “nah, humans make things up all the time!” But like, if you ask an LLM for someone’s biographical information, it sometimes will give highly specific fake details mixed in with real details, without being misled by unreliable sources or having an agenda to persuade you of. Even an overconfident and dishonest human wouldn’t do that. There’s clearly something different in kind from what we humans do.
The resilience of AI capabilities denialism is wild. I’m not proud to dunk on an internet rando, but I can’t stop thinking about this comment from a Substack thread on OpenAI solving the Navier-Stokes problem:
Two years ago, even at my most cynical, I don’t think I could’ve predicted that people would still be crying stochastic parrot in response to AI solving a Millennium Prize problem.
The words formed by a stack of compartmentalized biochemistry are just a result of chemical and electrical signal passing. They wouldn’t come into existence if you halved or doubled the amount of coffee on that day or presented it with slightly different social media
promptsposts. Nearly 100% of the time the output is absurd.the wise man bowed his head solemnly and spoke: “theres actually zero difference between good & bad things. because bad things are over-determined, and contributing to them makes little difference on the margin, and therefore the biggest counterfactual impact comes from steering them towards being good. you imbecile. you fucking moron”