TL;DR: Recently I wrote a post that got much less karma than I expected. My best guess is that the main reason is that I translated it from Russian to English via ChatGPT, and this easily recognizable LLM-style convinced readers from the first lines that “There aren’t any original thoughts; it is a standard, machine-generated fluff piece”. Is it correct, or does the post itself contain any major issues?
More detailed thoughts:
I expected the ideas from this post to be quite non-trivial (the math is simple, but practical results are more or less interesting; I mean, even Bayes’ and Aumann’s theorems are very simple from the mathematical point of view, but they bring many useful insights).
When I read it in the mentioned book, I found it very interesting.
When I published the original Russian post in my Telegram channel, it became one of the most liked posts there.
It is not easy to see it from the text, but I added many personal insights of my own to the book’s ideas (most of them are “how exactly it goes from the math” and “why it is still working in real life”).
I expected—including after considering my previous post—a rating of about 10-20 (and if I were an author who has been writing for a long time and proved his posts as useful for community, maybe I could expect 50+). Rating of 5 (2 of which are default rating based on author’s karma) makes me feel very confused.
My main hypothesis is as follows: the user opened the post, read one or two paragraphs, noticed distinct LLM-generated style, understood that it’s just another empty machine-generated piece of text, closed the post.
Part of the effect may be due to my not adding any tags to the post. But I don’t expect it was the main reason.
Perhaps the post itself is not well written or lacks new ideas; but my evals of such effect are already contained in my expected 10-20 rating.
And now, more detailed questions:
Are my assumptions correct? More specifically, I have the following questions (depending on how much effort you are willing to put in):
If you see text, which is obviously LLM-written/translated, what is the probability that you would stop reading after one or two paragraphs (except in cases where there is quite a revelation within these paragraphs)?
How strong is feeling from first one or two paragraphs of this particular post, that “There is nothing interesting in here; I will barely update; it’s just another LLM-written post”?
If you read the whole post—how much information from it is interesting? Are there any significant flaws in its content? How unpleasant is reading such LLM-style text?
Assuming that the main problem is just LLM-translation, what should I do next time? Should I translate it manually (as I did for this particular question-post—unfortunately, it took significantly more effort, and I’m not sure about its quality)? Or maybe I should add a disclaimer at the beginning of a post: “This text is completely thought through and written by myself, but translated by an LLM with minimal editing”? Or should I do something else?
I looked at your post:
I don’t think I ever saw it on the front page. If I did, I don’t think the title would have made it seem worth reading.
Reading the first few paragraphs: I’m not sure who this post is for, what problem is it addressing or why should I care in general. It’s referencing some specific chapter out of a book I never heard of.
It seems like a long post, so I just skip reading it.
I hope the suggestions are easy to derive from the above points.
The writing is not amazing but I didn’t get LLM vibes.
After reading the post, it felt very academic, or like an old book. Something about it makes me wonder if the LLM was too literal in its translation. I did an experiment translating Korean light novels with an LLM, and the result was also an uncanny valley where the translation was obviously correct but still hard to read.
I wonder if this has something to do with how as we read, we’re also trying to predict what the word will be. If my internal predictor is looking for the sentence that would come next in a normal English text but instead gets the sentence that would come next in a Russian text, translated to English, I’m constantly thrown off and it’s hard to read.
For what it’s worth, it doesn’t read like something written by an LLM.
I wonder if giving it an example of the intended translated writing style helps.
I don’t remember seeing it, and based on the title I probably wouldn’t have clicked. I’m not sure what’s wrong with the title but it feels kind of like a meaningless string of words at first glance (did you use an LLM to translate or create the title?). Some titles that feel more interesting/meaningful:
Why We Resist Improvement
Why Making Things Better Often Sucks
Resisting Improvement
Stepping Away From the Local Maximum
As for the article itself, it feels strangely hard to read to me, even if I don’t recognize it as LLM generated explicitly. Like my attention just keeps slipping away while trying to read it. This is a feeling I often get from text written by LLMs, especially text not generated at my behest. Nothing in this post had the same feeling. So I think it’s probably still worth translating things you want people to read by hand; it might be interesting to post a manual translation of the same article in a month or so to see how it does.
There are probably still plenty of ways you can use LLMs to speed up or enhance the process, e.g.
Have it generate 5 different translations of a sentence, then mix and match your favorite parts of each translation.
Do a rough translation yourself and then ask the LLM to point out places where it’s awkward, or has incorrect grammar.
Ask the LLM about the connotations of specific word choices.
The idea itself I found somewhat interesting, and probably could find it more interesting/useful with the right framing. I agree that 10-20 is a reasonable expectation based on just the ideas.
All the LLM’s know what LessWrong happens to be. After you asked the LLM to translate the post, ask it:
A) I want to post this on LessWrong. How do you think LessWrong readers will receive the post? What do you think the biggest issues of the post happen to be for LessWrong readers? What do you think LessWrong readers like most about the post.
After you hear the concerns from the LLM, you might have ideas about how to address the concerns. If so, tell the LLM your ideas about how to address the concerns and work it into the text. If you don’t have ideas yourself about how to address the concerns, you can also ask the LLM for ideas.
B) Would you make any stylistic changes to the post, so that it matches better the expectations of readers of LessWrong?
C) How would [your favorite writer on LessWrong] write this post?
If you come up with any issues about the post yourself, you can also ask the LLM about those issues.
The original title in Russian is “Насколько сложно менять(ся)”. This is a great title. “Насколько сложно” means “How difficult/hard is it” and “менять(ся)” means “to change (onesself)”.
Translated directly into English, the title should be “How difficult is it to change (onesself)”. This is a worse title than the original Russian, but not a terrible title.
The title you posted, “The Structure of the Pain of Change”, is awful. It’s unclear to me [from the title alone] what the post is supposed to be about. It gives Literary Theory vibes, and not in a good way.
Good titles are really important. I don’t know if this title was LLM-generated, and I cannot tell just from the title itself. To me, the problem isn’t that the post smells of LLM generation. The problem is that the title is mush.
If you are relying heavily on LLM-aided translation, then there should be a disclaimer at the top. This wouldn’t necessarily turn readers off. I’m often curious what’s happening in other languages. “Here is the Russian intellectual dialogue you’re missing out on” is a good hook.
Assuming you’re proficient at English, you should review the output and check its standalone quality.
Don’t worry about the tags. They rarely make or break a post.