Should you write longer comments? (Statistical analysis of the relationship between comment length and ratings)

A few months ago we have launched an ex­per­i­men­tal web­site. In brief, our goal is to cre­ate a plat­form where un­re­stricted free­dom of speech would be com­bined with high qual­ity of dis­cus­sion. The prob­lem can be ap­proached from two di­rec­tions. One is to help users nav­i­gate through con­tent and quickly lo­cate the higher qual­ity posts. Another, which is the topic of this ar­ti­cle, is to help users im­prove the qual­ity of their own posts by pro­vid­ing them with mean­ingful feed­back.

One im­por­tant con­sid­er­a­tion for those who want to write bet­ter com­ments is how much de­tail to leave out. Our statis­ti­cal anal­y­sis shows that for many users there is a strong con­nec­tion be­tween the rat­ings and the size of their com­ments. For ex­am­ple, for Yvain (Scott Alexan­der) and Eliezer_Yud­kowsky, the av­er­age num­ber of up­votes grows al­most lin­early with in­creas­ing com­ment length.

This trend, how­ever, does not ap­ply to all posters. For ex­am­ple, for the group of top ten con­trib­u­tors (in the last 30 days) to LessWrong, the av­er­age num­ber of up­votes in­creases only slightly with the length of the com­ment (see the graph be­low). For quite a few peo­ple the change even goes in the op­po­site di­rec­tion – longer com­ments lead to lower rat­ings.

Nat­u­rally, even if your longer com­ments are rated higher than the short ones, this does not mean that in­flat­ing com­ments would always pro­duce pos­i­tive re­sults. For most users (in­clud­ing pop­u­lar writ­ers, such as Yvain and Eliezer), the av­er­age num­ber of down­votes in­creases with in­creas­ing com­ment length. The data also shows that long com­ments that get most up­votes are gen­er­ally dis­tinct from long com­ments that get most down­votes. In other words, long com­ments are fine as long as they are in­ter­est­ing, but they are pe­nal­ized more when they are not.

The rat­ing pat­terns vary sig­nifi­cantly from per­son to per­son. For some posters, the av­er­age num­ber of up­votes re­mains flat un­til the com­ment length reaches some thresh­old and then starts de­clin­ing with in­creas­ing com­ment length. For oth­ers, the op­ti­mal com­ment length may be some­where in the mid­dle. (Users who have ac­counts on both Less­wrong and Om­nilibrium can check the op­ti­mal length for their own com­ments on both web­sites by us­ing this link.)

Ob­vi­ously length is just one among many fac­tors that af­fect com­ment qual­ity and for most users it does not ex­plain more than 20% of vari­a­tion in their rat­ings. We have a few other ideas on how to provide peo­ple with mean­ingful feed­back on both the style and the con­tent of their posts. But be­fore im­ple­ment­ing them, we would like to get your opinions first. Would such feed­back be ac­tu­ally use­ful to you?