I really enjoyed reading your AI companion piece, Zvi. Some LessWrong users are probably tired of seeing LLM sycophancy discussed, but it’s truly one of my favorite subjects in AI ethics. I think it would be worth doing a deeper dive into the distinct differences between overt persuasion with intent and a stochastic LLM being sycophantic.
For example, in my inaugural (albeit slightly cringe) LessWrong post, I discussed my experience with AI companion software, and how my background primed me to be more vulnerable to LLM praise.. With a model such as GPT4o or even O3, the psychological risk is more evident in being flattered, rather than being outright persuaded to change an opinion. When agents go from their current 2025 “stumbling” state to a more true autonomous entity, or even AGI, we won’t know with the same certainty what the goals are of the AI in question. Would a “true AGI” companion continue to flatter and praise its user with docility and kindness, or try to use more nefarious methods to get a person to be aligned with its goals?
Additionally, there is a real and present danger in the “male loneliness pandemic”, especially for autistic teenagers and young men, who have 7x higher chances of dying by suicide compared to the neurotypical population. On the one hand, a predatory AI lab or Big Tech firm could capitalize on this higher propensity and market their companion software to this vulnerable group (Elon could very well be doing this right now with “Ani” and XAI’s Grok 4 model), or the companion could be a net good, therapeutic product to help alleviate the very real harm of loneliness.
I believe the more immediate risk currently lies in excessive affirmation and praise from LLMs and less sophisticated AI agents rather than overt persuasion, though the latter becomes genuinely concerning when intentionally guided by human actors. Additionally, moral panics around human sexuality involving lonely men (autistic or neurotypical) engaging with AI companions or the popularity of VTubers and idol culture among women tend to be both unproductive and exaggerated.
I really enjoyed reading your AI companion piece, Zvi. Some LessWrong users are probably tired of seeing LLM sycophancy discussed, but it’s truly one of my favorite subjects in AI ethics. I think it would be worth doing a deeper dive into the distinct differences between overt persuasion with intent and a stochastic LLM being sycophantic.
For example, in my inaugural (albeit slightly cringe) LessWrong post, I discussed my experience with AI companion software, and how my background primed me to be more vulnerable to LLM praise.. With a model such as GPT4o or even O3, the psychological risk is more evident in being flattered, rather than being outright persuaded to change an opinion. When agents go from their current 2025 “stumbling” state to a more true autonomous entity, or even AGI, we won’t know with the same certainty what the goals are of the AI in question. Would a “true AGI” companion continue to flatter and praise its user with docility and kindness, or try to use more nefarious methods to get a person to be aligned with its goals?
Additionally, there is a real and present danger in the “male loneliness pandemic”, especially for autistic teenagers and young men, who have 7x higher chances of dying by suicide compared to the neurotypical population. On the one hand, a predatory AI lab or Big Tech firm could capitalize on this higher propensity and market their companion software to this vulnerable group (Elon could very well be doing this right now with “Ani” and XAI’s Grok 4 model), or the companion could be a net good, therapeutic product to help alleviate the very real harm of loneliness.
I believe the more immediate risk currently lies in excessive affirmation and praise from LLMs and less sophisticated AI agents rather than overt persuasion, though the latter becomes genuinely concerning when intentionally guided by human actors. Additionally, moral panics around human sexuality involving lonely men (autistic or neurotypical) engaging with AI companions or the popularity of VTubers and idol culture among women tend to be both unproductive and exaggerated.