I’ve found LLMs pretty useful for giving feedback on writing, including writing a fairly complex philosophical piece. Recently I wondered, “Hm, is this evidence that LLMs’ capabilities can generalize well to hard-to-verify tasks? That would be an update for me (toward super-short timelines, for one).”
I haven’t thought deeply about this yet, but I think the answer is: no. We should disentangle the intuitive “messiness” of the task of giving writing advice, from how difficult success is to verify:
Yes, the function in my brain that maps “writing feedback I’m given” to “how impressed I am with the feedback” seems pretty messy. My evaluation function for writing feedback is more fuzzy than, say, a function that checks if some piece of code works. It’s genuinely cool that LLMs can learn the former, especially without fine-tuning on me in particular.
But it’s still easy to verify whether someone has a positive vs. negative reaction[1] to some writing feedback. The model doesn’t need to actually be good at (1) making the thing I’m writing more effective at its intended purposes — i.e., helping readers have more informed and sensible views on the topic — in order to be good at (2) making me think “yeah that suggestion seems reasonable.”
Presumably there’s some correlation between the two. Humans rely on this correlation when giving writing advice all the time. But the correlation could still be rather weak (and its sign rather context-dependent), and LLMs’ advice would look just as impressive to me.
Maybe this is obvious? My sense, though, is that these two things could easily get conflated.
“Reaction” is meant to capture all the stuff that makes someone consider some feedback “useful” immediately when (or not too long after) they read it. I’m not saying LLMs are trying to make me feel good about my writing in this context, though that’s probably true to some extent.
“Messy” tasks vs. hard-to-verify tasks
(Followup to here.)
I’ve found LLMs pretty useful for giving feedback on writing, including writing a fairly complex philosophical piece. Recently I wondered, “Hm, is this evidence that LLMs’ capabilities can generalize well to hard-to-verify tasks? That would be an update for me (toward super-short timelines, for one).”
I haven’t thought deeply about this yet, but I think the answer is: no. We should disentangle the intuitive “messiness” of the task of giving writing advice, from how difficult success is to verify:
Yes, the function in my brain that maps “writing feedback I’m given” to “how impressed I am with the feedback” seems pretty messy. My evaluation function for writing feedback is more fuzzy than, say, a function that checks if some piece of code works. It’s genuinely cool that LLMs can learn the former, especially without fine-tuning on me in particular.
But it’s still easy to verify whether someone has a positive vs. negative reaction[1] to some writing feedback. The model doesn’t need to actually be good at (1) making the thing I’m writing more effective at its intended purposes — i.e., helping readers have more informed and sensible views on the topic — in order to be good at (2) making me think “yeah that suggestion seems reasonable.”
Presumably there’s some correlation between the two. Humans rely on this correlation when giving writing advice all the time. But the correlation could still be rather weak (and its sign rather context-dependent), and LLMs’ advice would look just as impressive to me.
Maybe this is obvious? My sense, though, is that these two things could easily get conflated.
“Reaction” is meant to capture all the stuff that makes someone consider some feedback “useful” immediately when (or not too long after) they read it. I’m not saying LLMs are trying to make me feel good about my writing in this context, though that’s probably true to some extent.