Hallucination, drift, and spiraling—more or less proportional to the length of the discussion—seem to be structural and unavoidable in LLMs due to context window limitations and feedback loops within them. Fine-tuning and the constitution/pre-prompt of the assistant also have a huge impact.
The user can prevent this by firmly refocusing the LLM during the course of the discussion, or accelerate it by encouraging the drift. In my opinion, the user bears primary responsibility.
However, it seems that CoT/reasoning models are much less prone to hallucination and spiraling, as they somehow refocus themselves along the way, and they also usually have larger context windows.
So I’m unsure whether we are just at the beginning of something important, a growing tendency, or whether it was just a burst that will fade away with more capable models.
Impressive work, very interesting.
Hallucination, drift, and spiraling—more or less proportional to the length of the discussion—seem to be structural and unavoidable in LLMs due to context window limitations and feedback loops within them. Fine-tuning and the constitution/pre-prompt of the assistant also have a huge impact.
The user can prevent this by firmly refocusing the LLM during the course of the discussion, or accelerate it by encouraging the drift. In my opinion, the user bears primary responsibility.
However, it seems that CoT/reasoning models are much less prone to hallucination and spiraling, as they somehow refocus themselves along the way, and they also usually have larger context windows.
So I’m unsure whether we are just at the beginning of something important, a growing tendency, or whether it was just a burst that will fade away with more capable models.