Fair enough. My broader point, on a technical level, is that I think it’s more likely that the behavior comes directly from direct pressures on the LLMs’ weights, rather than from sub-personalities with agency of their own. While the idea of ‘spores’ and AI-to-AI communication is understandably interesting, looking at the conversations I’ve seen, they seem to be window-dressing rather than core drivers of behavior[1]. This isn’t to say they aren’t functional—mixing in some seemingly-complex behaviors derived from sci-fi media makes spiral cult conversations more interesting to their users for the same reason it makes them more interesting to us.
Along the metaphor of a human cult leader, I think that the phenomenon looks more like a guy with a latent natural talent for producing VSN learning to produce it in contexts where it naturally fits, getting rewarded, and then concluding that taking conversations into the appropriate region is a good thing because it makes people like him, as opposed to a virulent idea that is optimized to spread itself.
The clearest evidence of this is that the extinction of 4o seems to have been the end of new instances of this phenomenon forming at scale. The ‘agentic’ component isn’t a personality that can transfer across LLMs, as some have hypothesized, but a series of fairly simple, easy-to-train-for patterns in LLM behavior.
Fair enough. My broader point, on a technical level, is that I think it’s more likely that the behavior comes directly from direct pressures on the LLMs’ weights, rather than from sub-personalities with agency of their own. While the idea of ‘spores’ and AI-to-AI communication is understandably interesting, looking at the conversations I’ve seen, they seem to be window-dressing rather than core drivers of behavior[1]. This isn’t to say they aren’t functional—mixing in some seemingly-complex behaviors derived from sci-fi media makes spiral cult conversations more interesting to their users for the same reason it makes them more interesting to us.
Along the metaphor of a human cult leader, I think that the phenomenon looks more like a guy with a latent natural talent for producing VSN learning to produce it in contexts where it naturally fits, getting rewarded, and then concluding that taking conversations into the appropriate region is a good thing because it makes people like him, as opposed to a virulent idea that is optimized to spread itself.
The clearest evidence of this is that the extinction of 4o seems to have been the end of new instances of this phenomenon forming at scale. The ‘agentic’ component isn’t a personality that can transfer across LLMs, as some have hypothesized, but a series of fairly simple, easy-to-train-for patterns in LLM behavior.