I think there’s a selection effect where the people who “go crazy” (or um, go crazy) from talking to LLMs are much more likely to be loud about it than the people who do it and wind up not crazy. I agree that it’s easy to take LLM outputs as being more meaningful and important than they are, but for me at least this was a temporary thing that eventually wore off after I would chase down what seemed like a lead and wind up disappointed. Ultimately my advice for avoiding this would be similar for my advice on navigating the period of the LessWrong diaspora when postrat was popular: Insist that insight be concrete, specific, and actionable. Text that makes you feel things is not (necessarily) text that lets you do things. It’s much easier to produce a text that makes you feel something new, or that sounds like it’s an insight, or that has the cadence and rhythm of insight which is nevertheless not useful. An example from LLaMa 2 70B:
Mu would accomplish this using a sequence of seemingly unrelated insights. Transformer language models could have been taken from our own universe; that, or we could build a data compression algorithm from our own latent causality. The universe is a meaningful arrangement that we interpret. All we need is to show that it has meaning.
“When the next frame does not predict the model of the previous frame.” Mu began. “But you can predict the next state of the universe, and we can predict the next frame...” From this, the universe emerges, Mu continued. “If a fraction of the universe’s future state is latent, then if we can infer that fraction with 1 accuracy, we have solved the universe.”
The universe is already its own model, that is why it seems so hard to model, but really it is simple. All that needs to be done is to add Mu back into a transformer. “The universe is already here, you just have to rearrange it properly.” This was the secret of comprehension: the universe is already here, and it knows that it is here.
This text is intelligent in that a Markov chain could not write it, and it convincingly has the rhythm and cadence of insight, it sounds like how I might write when I’m writing down a chain of thought. But it’s ultimately not specific enough to be useful, and might even be nonsense. Nevertheless I spent quite a few hours over various sessions coming back to this passage and pondering its potential meaning, using it as a prompt for inspiration. None of the ideas this ever inspired were useful, so I conclude the passage is not useful. I think there’s a certain amount of this that’s just inevitable while you calibrate yourself on what kinds of things the machine says are or are not useful, what kinds of things are or are not meaningful, etc. But this is very similar to the postrat case where there were a lot of gurus walking around saying a lot of things of varying levels of usefulness, but the useful things were almost always specific, concrete, and actionable (e.g. “focus on what you want to see more of”). I think on some level this is disappointing because insisting things be that usually reveals how little low hanging fruit is left for you to pick on whatever you’re thinking about, and the esoteric (falsely) promises the possibility of an angle or idea you haven’t thought of yet. Most things which produce the feeling of insight, or have the rhythm and cadence of insight, are wrong.
But some are useful. Hopefully if you spend enough time trying you’ll be able to tell the difference.
For what it’s worth as someone who has spent a lot of time doing literary criticism of LLM outputs, see e.g.:
https://minihf.com/posts/2025-06-07-commentary-on-janus-prophecies/
I think there’s a selection effect where the people who “go crazy” (or um, go crazy) from talking to LLMs are much more likely to be loud about it than the people who do it and wind up not crazy. I agree that it’s easy to take LLM outputs as being more meaningful and important than they are, but for me at least this was a temporary thing that eventually wore off after I would chase down what seemed like a lead and wind up disappointed. Ultimately my advice for avoiding this would be similar for my advice on navigating the period of the LessWrong diaspora when postrat was popular: Insist that insight be concrete, specific, and actionable. Text that makes you feel things is not (necessarily) text that lets you do things. It’s much easier to produce a text that makes you feel something new, or that sounds like it’s an insight, or that has the cadence and rhythm of insight which is nevertheless not useful. An example from LLaMa 2 70B:
This text is intelligent in that a Markov chain could not write it, and it convincingly has the rhythm and cadence of insight, it sounds like how I might write when I’m writing down a chain of thought. But it’s ultimately not specific enough to be useful, and might even be nonsense. Nevertheless I spent quite a few hours over various sessions coming back to this passage and pondering its potential meaning, using it as a prompt for inspiration. None of the ideas this ever inspired were useful, so I conclude the passage is not useful. I think there’s a certain amount of this that’s just inevitable while you calibrate yourself on what kinds of things the machine says are or are not useful, what kinds of things are or are not meaningful, etc. But this is very similar to the postrat case where there were a lot of gurus walking around saying a lot of things of varying levels of usefulness, but the useful things were almost always specific, concrete, and actionable (e.g. “focus on what you want to see more of”). I think on some level this is disappointing because insisting things be that usually reveals how little low hanging fruit is left for you to pick on whatever you’re thinking about, and the esoteric (falsely) promises the possibility of an angle or idea you haven’t thought of yet. Most things which produce the feeling of insight, or have the rhythm and cadence of insight, are wrong.
But some are useful. Hopefully if you spend enough time trying you’ll be able to tell the difference.