Have you tried running LLMs on this sophisticated LLM-slop, asking them to determine whether there are any meaningful contributions, rather than this being just slop?
This approach has its perils, of course, both now (AFAIK, LLMs already tend to prefer LLM-generated content to human-generated content), and in the future, but I would expect this to be somewhat non-trivially useful.
Have you tried running LLMs on this sophisticated LLM-slop
Yeah, I just started experimenting with this yesterday (originally I put in a footnote in the OP and for some reason removed it).
Yesterday what I tried was making a “Simulated JohnWentworth” review bot, and a “Simulated NeelNanda” review bot. I only tested it so far on ~3 new-user-slop-posts, ~2 new-user-ambiguous-maybe-slop-posts, a couple Best of All Time posts, and some random posts from the last month.
It did okay-ish. (The Wentworth one definitely had Wentworth-esque criticisms of like “you are making up words and thinking you have made progress but you haven’t shown me that you can distinguish the symbol from the real thing you’re talking about or break it down into gears.” But, it applied them kinda indiscriminately in a way I didn’t particularly trust)
In theory one would not want to use LLMs for big judgements like this, but rather for making simpler judgements that they should be much better at, e.g. classifying topics or something more sophisticated like projecting into a medium-dimensional space which is then used for metafiltering. E.g. a space in which human users also have vectors which indicate “this person is this much of an expert in this direction” or something like that, where “expert” functionally means stuff like gets a stronger vote on posts like that. (This is sort of a crowdsourced moderation scheme, but trying to be more efficient in the way that current moderators are more efficient than random users having to downvote slop, but a bit more distributed.)
Have you tried running LLMs on this sophisticated LLM-slop, asking them to determine whether there are any meaningful contributions, rather than this being just slop?
This approach has its perils, of course, both now (AFAIK, LLMs already tend to prefer LLM-generated content to human-generated content), and in the future, but I would expect this to be somewhat non-trivially useful.
Yeah, I just started experimenting with this yesterday (originally I put in a footnote in the OP and for some reason removed it).
Yesterday what I tried was making a “Simulated JohnWentworth” review bot, and a “Simulated NeelNanda” review bot. I only tested it so far on ~3 new-user-slop-posts, ~2 new-user-ambiguous-maybe-slop-posts, a couple Best of All Time posts, and some random posts from the last month.
It did okay-ish. (The Wentworth one definitely had Wentworth-esque criticisms of like “you are making up words and thinking you have made progress but you haven’t shown me that you can distinguish the symbol from the real thing you’re talking about or break it down into gears.” But, it applied them kinda indiscriminately in a way I didn’t particularly trust)
In theory one would not want to use LLMs for big judgements like this, but rather for making simpler judgements that they should be much better at, e.g. classifying topics or something more sophisticated like projecting into a medium-dimensional space which is then used for metafiltering. E.g. a space in which human users also have vectors which indicate “this person is this much of an expert in this direction” or something like that, where “expert” functionally means stuff like gets a stronger vote on posts like that. (This is sort of a crowdsourced moderation scheme, but trying to be more efficient in the way that current moderators are more efficient than random users having to downvote slop, but a bit more distributed.)