I agree with most points on a first pass, but I’m still unsure about:
you must have added significant value beyond what the AI produced
Shouldn’t the target for posts be to provide value? If an entirely AI-generated post passes every quality check and appears to be on equal footing to a human post in terms of value, I’d want it. Attribution of credit is a valid concern, but it seems like the solution there is to simply tag the model as the primary author.
In theory, maybe. In practice, people who can’t write well usually can’t discern well either, and the LLM submissions that are actually submitted to LW have much lower average quality than the human-written posts. Even if they were of similar quality, they’re still drawn from a different distribution, and the LLM-distribution is one that most readers can draw from if they want (with prompts that are customized to what they want), while human-written content is comparatively scarce.
IMO a not yet fully understood but important aspect of this situation is that what someone writes is in part testimony—they’re asserting something that others may or may not be able to verify themselves easy, or even at all. This is how communication usually works, and it has goods (you get independent information) and bads (people can lie/distort/troll/mislead). If a person is posting AIgen stuff, it’s much less so testimony from that person. It’s more correlated with other stuff that’s already in the water, and it’s not revealing as much about the person’s internal state—in particular, their models. I’m supposed to be able to read text under the presumption that a person with a life is testifying to the effect of what’s written. Even if you go through and nod along with what the gippity wrote, it’s not the same. I want you to generate it yourself from your models so I can see those models, I want to be able to ask you followup questions, and I want you to stake something of the value of your word on what you publish. To the extent that you might later say “ah, well, I guess I hadn’t thought XYZ through really, so don’t hold me to account for having apparently testified to such; I just got a gippity to write my notions up quickly”, then I care less about the words (and they become spammier).
(note: This is Raemon’s random take rather than considered Team Consensus)
Part of the question here is “what sort of engine is overall maintainable, from a moderation perspective?”.
LLMs make it easy for tons of people to be submitting content to LessWrong without really checking whether it’s true and relevant. It’s not enough for a given piece to be true. It needs to be reliably true, with low cost to moderator attention.
Right now, basically LLMs don’t produce anywhere near good enough content. So, presently, letting people submit AI generated content without adding significant additional value is a recipe for LW admins to spend a bunch of extra time each day deciding whether to moderate a bunch of content that we’re realistically going to say “no” to.
(Some of the content is ~on par with the bottom 25% of LW content, but the bottom 25% of LW content is honestly below the quality bar we prefer the site to be at, and the reason we let those comments/posts in at all is because it’s too expensive to really check if it’s reasonable, and when we’re unsure, we sometimes to default to “let it in, and let the automatic rate limits handle it”. But, the automated rate limits would not be sufficient to handle an influx of LLM slop)
But, even when we imagine content that should theoretically be “just over the bar”, there are secondorder effects of LW being a site with a potentially large amount of AI content that nobody is really sure if it’s accurate or whether anyone endorses it and whether we are entering into some slow rolling epistemic disaster.
So, my guess for the bar for “how good quality do we need to be talking about for AI content to be net-positive” is more at least top-50% and maybe top-25% of baseline LW users. And when we get to that point probably the world looks pretty different.
I agree with most points on a first pass, but I’m still unsure about:
Shouldn’t the target for posts be to provide value? If an entirely AI-generated post passes every quality check and appears to be on equal footing to a human post in terms of value, I’d want it. Attribution of credit is a valid concern, but it seems like the solution there is to simply tag the model as the primary author.
In theory, maybe. In practice, people who can’t write well usually can’t discern well either, and the LLM submissions that are actually submitted to LW have much lower average quality than the human-written posts. Even if they were of similar quality, they’re still drawn from a different distribution, and the LLM-distribution is one that most readers can draw from if they want (with prompts that are customized to what they want), while human-written content is comparatively scarce.
IMO a not yet fully understood but important aspect of this situation is that what someone writes is in part testimony—they’re asserting something that others may or may not be able to verify themselves easy, or even at all. This is how communication usually works, and it has goods (you get independent information) and bads (people can lie/distort/troll/mislead). If a person is posting AIgen stuff, it’s much less so testimony from that person. It’s more correlated with other stuff that’s already in the water, and it’s not revealing as much about the person’s internal state—in particular, their models. I’m supposed to be able to read text under the presumption that a person with a life is testifying to the effect of what’s written. Even if you go through and nod along with what the gippity wrote, it’s not the same. I want you to generate it yourself from your models so I can see those models, I want to be able to ask you followup questions, and I want you to stake something of the value of your word on what you publish. To the extent that you might later say “ah, well, I guess I hadn’t thought XYZ through really, so don’t hold me to account for having apparently testified to such; I just got a gippity to write my notions up quickly”, then I care less about the words (and they become spammier).
(note: This is Raemon’s random take rather than considered Team Consensus)
Part of the question here is “what sort of engine is overall maintainable, from a moderation perspective?”.
LLMs make it easy for tons of people to be submitting content to LessWrong without really checking whether it’s true and relevant. It’s not enough for a given piece to be true. It needs to be reliably true, with low cost to moderator attention.
Right now, basically LLMs don’t produce anywhere near good enough content. So, presently, letting people submit AI generated content without adding significant additional value is a recipe for LW admins to spend a bunch of extra time each day deciding whether to moderate a bunch of content that we’re realistically going to say “no” to.
(Some of the content is ~on par with the bottom 25% of LW content, but the bottom 25% of LW content is honestly below the quality bar we prefer the site to be at, and the reason we let those comments/posts in at all is because it’s too expensive to really check if it’s reasonable, and when we’re unsure, we sometimes to default to “let it in, and let the automatic rate limits handle it”. But, the automated rate limits would not be sufficient to handle an influx of LLM slop)
But, even when we imagine content that should theoretically be “just over the bar”, there are secondorder effects of LW being a site with a potentially large amount of AI content that nobody is really sure if it’s accurate or whether anyone endorses it and whether we are entering into some slow rolling epistemic disaster.
So, my guess for the bar for “how good quality do we need to be talking about for AI content to be net-positive” is more at least top-50% and maybe top-25% of baseline LW users. And when we get to that point probably the world looks pretty different.