In theory one would not want to use LLMs for big judgements like this, but rather for making simpler judgements that they should be much better at, e.g. classifying topics or something more sophisticated like projecting into a medium-dimensional space which is then used for metafiltering. E.g. a space in which human users also have vectors which indicate “this person is this much of an expert in this direction” or something like that, where “expert” functionally means stuff like gets a stronger vote on posts like that. (This is sort of a crowdsourced moderation scheme, but trying to be more efficient in the way that current moderators are more efficient than random users having to downvote slop, but a bit more distributed.)
In theory one would not want to use LLMs for big judgements like this, but rather for making simpler judgements that they should be much better at, e.g. classifying topics or something more sophisticated like projecting into a medium-dimensional space which is then used for metafiltering. E.g. a space in which human users also have vectors which indicate “this person is this much of an expert in this direction” or something like that, where “expert” functionally means stuff like gets a stronger vote on posts like that. (This is sort of a crowdsourced moderation scheme, but trying to be more efficient in the way that current moderators are more efficient than random users having to downvote slop, but a bit more distributed.)