I have a different argument in mind. The texts, like viruses, are to either stay underdeveloped piles of ideas or to be read by a mind and to cause a reaction. Upon reading a text, the humans would, for example, receive a new idea (and have the chance to spread it further) or change something else in their minds based on the text and the sender. The LLMs from your example would receive the ability to provide honest feedback (e.g. tell the human about books where the idea is disproven or developed further). Then the text and the LLM’s feedback are read by the human oneself[1] and cause a reaction only in the human’s mind.
The best-case scenario would have the humans use the LLMs in ways which improve the memetic environment by making human slop a bit less sloppy (e.g. by having the LLM give advice on better expression or criticise the least plausible ideas like the ones which are sent to LessWrong and rejected). The worse-case scenario would make the humans more self-confident[2] after receiving sycophantic feedback (e.g. from GPT-4o) and more likely to publish slop. And the absolute worst-case scenario is the one where the AI itself convinces the human to write the message and to release it into the wild or to do an act which any normal human would reject.
I have a different argument in mind. The texts, like viruses, are to either stay underdeveloped piles of ideas or to be read by a mind and to cause a reaction. Upon reading a text, the humans would, for example, receive a new idea (and have the chance to spread it further) or change something else in their minds based on the text and the sender. The LLMs from your example would receive the ability to provide honest feedback (e.g. tell the human about books where the idea is disproven or developed further). Then the text and the LLM’s feedback are read by the human oneself[1] and cause a reaction only in the human’s mind.
The best-case scenario would have the humans use the LLMs in ways which improve the memetic environment by making human slop a bit less sloppy (e.g. by having the LLM give advice on better expression or criticise the least plausible ideas like the ones which are sent to LessWrong and rejected). The worse-case scenario would make the humans more self-confident[2] after receiving sycophantic feedback (e.g. from GPT-4o) and more likely to publish slop. And the absolute worst-case scenario is the one where the AI itself convinces the human to write the message and to release it into the wild or to do an act which any normal human would reject.
To be precise, they also can influence the reward model or the LLM by entering the training data.
IIRC there was a post on LessWrong asking what the humans would likely do with an artificial source of self-confidence. But I cannot find it.