At the top level, I believe the key distinction is between LLM psychosis and misbeliefs[1]. By LLM psychosis, I mean distortedthinking due to LLM interactions, seemingly most often appearing in people who have existing mental health issues or risk factors for mental health issues[2]. By misbeliefs, I mean people believing ordinary falsehoods which an LLM has told them are true, in the absence of distorted thinking (of course people with LLM psychosis are believing falsehoods also).
There’s certainly a spectrum between LLM psychosis and misbeliefs. For example, people who have been told by an LLM that they’ve made a scientific breakthrough may get really excited and not sleep nearly enough, resulting in mania symptoms even if they don’t have a predisposition toward that. Still, I think the categories are distinct enough to be useful.
How do we most naturally subdivide the misbelief category? I’m not sure. Beliefs in scientific breakthroughs and beliefs in LLMs becoming conscious seem like two common subcategories[3] with somewhat different dynamics, but there are surely others. For example, I imagine that many people end up believing false things about themselves in a way that doesn’t fit the other subcategories and isn’t LLM psychosis. I’d love to hear ideas for other natural-seeming subcategories.
Scott Alexander’s ‘In Search Of AI Psychosis’ is probably the best thing written to date on LLM psychosis, although I think he underestimates the prevalence by a factor of something like 2–4x.
One way in which the difference between LLM psychosis and misbelief shows: a substantial number of people are under the impression that LLMs are authoritative sources of truth. They don’t know anything about ML, they know ChatGPT is a really big deal, and they haven’t heard about hallucinations. Under those circumstances, it’s clear that no distorted thinking is needed for them to believe the LLM is correct when it tells them they have an important breakthrough.
A tentative top-level ontology of people believing false things due to LLMs
While writing ‘Your LLM-assisted scientific breakthrough probably isn’t real’, I spent a bit of time thinking about how this issue overlaps with other cases in which people end up with false beliefs as a result of conversations with LLMs.
At the top level, I believe the key distinction is between LLM psychosis and misbeliefs[1]. By LLM psychosis, I mean distorted thinking due to LLM interactions, seemingly most often appearing in people who have existing mental health issues or risk factors for mental health issues[2]. By misbeliefs, I mean people believing ordinary falsehoods which an LLM has told them are true, in the absence of distorted thinking (of course people with LLM psychosis are believing falsehoods also).
There’s certainly a spectrum between LLM psychosis and misbeliefs. For example, people who have been told by an LLM that they’ve made a scientific breakthrough may get really excited and not sleep nearly enough, resulting in mania symptoms even if they don’t have a predisposition toward that. Still, I think the categories are distinct enough to be useful.
How do we most naturally subdivide the misbelief category? I’m not sure. Beliefs in scientific breakthroughs and beliefs in LLMs becoming conscious seem like two common subcategories[3] with somewhat different dynamics, but there are surely others. For example, I imagine that many people end up believing false things about themselves in a way that doesn’t fit the other subcategories and isn’t LLM psychosis. I’d love to hear ideas for other natural-seeming subcategories.
To borrow an awkward term from cognitive science
Scott Alexander’s ‘In Search Of AI Psychosis’ is probably the best thing written to date on LLM psychosis, although I think he underestimates the prevalence by a factor of something like 2–4x.
Or maybe LLM consciousness is a subtype of scientific breakthroughs?
Update: The Rise of Parasitic AI provides another interesting angle on this.
One way in which the difference between LLM psychosis and misbelief shows: a substantial number of people are under the impression that LLMs are authoritative sources of truth. They don’t know anything about ML, they know ChatGPT is a really big deal, and they haven’t heard about hallucinations. Under those circumstances, it’s clear that no distorted thinking is needed for them to believe the LLM is correct when it tells them they have an important breakthrough.