Boo votes, Yay NPS

TL;DR

Many votes on LW are “boos” and “yays”, and con­se­quently they aren’t very use­ful for de­ter­min­ing what is worth read­ing. A mod­ified ver­sion of a Net Pro­moter Score (NPS) on each post may provide a bet­ter met­ric for de­ter­min­ing read wor­thi­ness.

Motivation

It’s come up a cou­ple time in my re­cent com­ments that I’ve ex­pressed a the­ory that votes on LW, AF, and EAF are “boos” and “yays”. I have an idea about how we could do bet­ter as­sum­ing the pur­pose of votes is not to jeer and cheer but to provide in­for­ma­tion about the post, speci­fi­cally how much the post is worth read­ing, so I’m fi­nally writ­ing it up so oth­ers can, yes, boo or ap­plaud my effort, but more im­por­tantly so we might dis­cuss ways to im­prove the sys­tem. If you don’t like my pro­posal and agree we could do bet­ter than votes, I en­courage you to write up your ideas and share them.

So, there are many things votes could be for, but I view votes as a solu­tion to a prob­lem, so what’s the prob­lem votes are try­ing to solve? The num­ber one ques­tion I want an­swered about ev­ery post is some ver­sion of “should I read this?”. There’s sub­tly differ­ent ways to phrase this ques­tion: “is this worth en­gag­ing with?”, “should I read this care­fully or just skim it?”, “is this worth my time and en­ergy?”, etc.

I want a solu­tion to this prob­lem be­cause when I come to LW/​AF/​EAF ev­ery day I want a re­li­able sig­nal about what it’s worth me spend­ing my en­ergy en­gag­ing with (I gen­er­ally don’t want to just read, but also com­ment, dis­cuss, un­der­stand, grow). Right now votes don’t provide this to me, as I’ll ex­plain be­low, but they do provide other things. So keep in mind that my goal in this pro­posal is pri­mar­ily to solve the par­tic­u­lar prob­lem of “should I read this?” and not the many other prob­lems votes might be solu­tions to like “how to de­liver sim­ple pos­i­tive/​nega­tive feed­back?”, “how can I ex­press my plea­sure or dis­plea­sure with a post?”, “how do we de­ter­mine sta­tus within the fo­rum?”, or “how do we in­crease plat­form en­gage­ment?”. I don’t ig­nore these other pur­poses, but I take them as sec­ondary—and maybe there’s other pur­poses I for­got to list and so for­got to take into ac­count! The point be­ing I want it to be clear I’m mak­ing a pro­posal that’s try­ing to solve a par­tic­u­lar prob­lem, and if you com­plain “but wait, it doesn’t solve this other prob­lem” my re­sponse will be “yep, sure doesn’t”, so any dis­cus­sion of this sort should be sure to ex­plain why we should care about this other thing.

Okay, all that out of the way, let’s talk about votes, and then NPS.

Boo Votes

Up/​down vot­ing is very sim­ple and has a long his­tory on LW, thanks to its pres­ence on Red­dit (from which, if I re­call cor­rectly, the origi­nal fo­rum’s code­base was forked). It has a num­ber of nice fea­tures, and LW has made them nicer:

  • ev­ery­one knows how it works

  • it lets you ex­press your­self in two ways (un­like on Twit­ter where the only op­tion is to vote up some­thing, and a “down­vote” re­quires writ­ing your own tweet ex­press­ing dis­like)

  • the ag­gre­gate votes on a post can be used to gen­er­ate a user score (karma)

  • the user score can be used to me­ter ac­cess to var­i­ous site features

  • votes are pro­por­tional the sta­tus of users, as mea­sured by karma

And of course lots of pop­u­lar fo­rums of all sorts use votes: Face­book, Twit­ter, Red­dit, Tum­blr. Even when votes aren’t pre­sent some­thing like vot­ing is in the form of “re­acts” where a per­son can choose from a list of named images/​sounds/​etc. to ex­press some­thing and that some­thing gen­er­ally can in­clude a sim­ple vote (usu­ally us­ing a uni­ver­sally rec­og­nized vote re­act, like thumbs up/​down); cf. Slack, Dis­cord, most mas­sively mul­ti­player games, Twitch. So it would seem that peo­ple like votes a lot and they are used to some effect in lots of places.

Un­for­tu­nately for our pur­poses of try­ing to figure out “should I read this?”, most of what votes are do­ing is only in­di­rectly en­gaged with this ques­tion. Votes, es­pe­cially if we think of them as a de­gen­er­ate case of re­acts, are more used to ex­press an opinion on the con­tent than to de­ter­mine whether or not the con­tent is worth read­ing, and when there are two vot­ing op­tions they tend to be rounded off to down = boo and up = yay. If you have any doubts about this, just spend more time on so­cial me­dia and let me know if you still dis­agree in gen­eral, i.e. you dis­agree that most peo­ple do this, not that you don’t do this or your small group of friends don’t do this.

On that point of us­ing votes for some­thing else, it’s tempt­ing to think “hey, this is LW; we’re ra­tio­nal AF; we know bet­ter than to use votes as boos and yays”. To which I say “please, tell us more about how you’ve man­aged to cre­ate a com­mu­nity of perfectly ra­tio­nal agents”.

Jok­ing aside, my point is that I’ve been on the re­ceiv­ing end of all kinds of vot­ing pat­terns, so I’ve got­ten a chance to see how peo­ple use votes on LW. Fur­ther, I’ve talked to peo­ple about my posts (ei­ther in com­ments or el­se­where) and in some cases ex­plic­itly learned how they voted on my posts and why, and it’s lead me to a few con­clu­sions about how peo­ple use votes here.

  • Some­times votes are at­tempts to in­crease or de­crease visi­bil­ity of some­thing, re­gard­less of how some­one feels about what’s in a post or com­ment.

  • Some­times votes are a gen­uine ex­pres­sion of “you should/​shouldn’t read this”.

  • Most of­ten votes say “yay, I like this” or “boo, I don’t like this” in re­sponse to one of sev­eral thing:

    • like/​dis­like the author

    • like/​dis­like the sub­ject matter

    • like/​dis­like the content

    • like/​dis­like the presentation

The re­sult is what I con­sider a lot of vot­ing anoma­lies from the per­spec­tive of try­ing to an­swer the ques­tion “should I read this?”. Some claims of things I’ve seen (I won’t link spe­cific posts be­cause I don’t want to risk ap­ply­ing shame to any­one for what hap­pened to their post in the votes, and also it’s a lot of work to dig up all the ex­am­ples that caused me to form these be­liefs):

  • Low con­tent/​qual­ity posts voted highly be­cause peo­ple like the author

  • High con­tent/​qual­ity posts voted lowly be­cause peo­ple dis­like the author

  • Posts voted down for heresy, re­gard­less of quality

  • Posts voted up for ap­plause lights, re­gard­less of quality

My per­sonal ex­pe­rience is mainly with writ­ing hereti­cal posts of good qual­ity such that I get more up votes than down but also a lot of down votes (maybe 13 down and 23 up), and it caused me to pay more at­ten­tion to vot­ing pat­terns, en­gage more with low score posts, and try to figure out just what was go­ing on when posts got low scores that I gave up­votes. What I learned lead me to sur­mise what I’ve pre­sented above.

So votes seem to be largely used to sig­nal ap­proval and dis­ap­proval of posts, which I sug­gest is only weakly cor­re­lated with tel­ling me whether or not I should read a post. As a re­sult I ba­si­cally ig­nore votes and have to skim ev­ery­thing to figure out where the good stuff is. But what if we could do some­thing bet­ter...?

Yay NPS

Net Pro­moter Score (NPS) is a sim­ple metic many com­pa­nies use to eval­u­ate ques­tions of cus­tomer satis­fac­tion. To calcu­late it peo­ple are asked “how likely are you to recom­mend our product or ser­vice to a friend or col­league?” and asked for a num­ber from 0 to 10, 0 mean­ing “not likely at all” and 10 mean­ing “already have”. I re­ally like NPS be­cause it asks peo­ple to imag­ine recom­mend­ing some­thing and then ask­ing them for some­thing like a prob­a­bil­ity of how likely they are to do it, al­though I’ve never seen a ver­sion that did this ex­plic­itly.

Re­sponses are then con­verted into a score by first seg­ment­ing re­spon­dents into de­trac­tors, pas­sives, and pro­mot­ers, and then tak­ing per­cent pro­mot­ers minus per­cent de­trac­tors. I find this met­ric to be of limited value, and more pre­fer to en­gage di­rectly with the full dis­tri­bu­tion of re­sponses, but if you re­ally needed a sin­gle scalar this is one way to get it.

What I imag­ine do­ing is ask­ing peo­ple to score posts like this:

How likely are you to recom­mend a friend or col­league read this post?

|--0%-------------50%-------------100%--|

So they are asked the ques­tion and given a slider to mark their like­li­hood, which in­cludes 100% be­cause they may have already shared it (but there’s prob­a­bly some UI work here to make it clear that 100% and 99% are dras­ti­cally differ­ent re­sponses).

Does this an­swer our ques­tion “should I read this?”? I think it may do a bet­ter job than votes, to be sure. Rather than an am­bigu­ous vote, peo­ple are now at least be­ing asked to re­spond di­rectly to a ques­tion and give their re­sponse to it. Also, we could bet­ter use the dis­tri­bu­tion of re­sponses to make read­ing de­ci­sions. For ex­am­ple, hereti­cal posts might get bi­modal dis­tri­bu­tions of scores, with clusters of strong de­trac­tors and strong pro­mot­ers, and maybe you choose to read a post when it has at least n pro­mot­ers, re­gard­less of de­trac­tors. Maybe you choose to filter out posts with more than n de­trac­tors be­cause you don’t like con­tro­versy or low qual­ity con­tent. Maybe you filter on NPS or mean or me­dian or some­thing else, or sort based on it. And ev­ery post, rather than show­ing a sim­ple num­ber for its score like we do now you show a box-plot or some other suit­able vi­su­al­iza­tion show­ing the dis­tri­bu­tion of re­sponses.

Now un­for­tu­nately NPS is more com­pli­cated than votes, so it may work against other prob­lems peo­ple are try­ing to solve with votes. How does NPS help us deal with the prob­lems ad­dressed by karma? How do we pre­vent NPS from de­volv­ing into a bi­nary where peo­ple always vote 100% to up­vote and ev­ery­thing else is a down­vote (the eBay/​Uber/​Lyft vot­ing prob­lem, where any­thing less than 5-stars is con­sid­ered a down­vote)? And do we mea­sure com­ment qual­ity with NPS, or keep votes there, or do some­thing else?

I also don’t re­ally ex­pect the LW team to drop ev­ery­thing and im­ple­ment NPS. Heck, if I were work­ing on LW I prob­a­bly wouldn’t jump all over this. My goal in writ­ing this, maybe more than any­thing, is to get us think­ing about how to bet­ter an­swer the ques­tion “should I read this?” and I wanted to provide at least one solu­tion I’ve thought of and think could be bet­ter in some ways. I mostly think we could do more to give bet­ter sig­nals of qual­ity on LW and make them less dis­torted by and en­gaged with other sig­nals peo­ple try to send with votes.

So, what do you think of the cur­rent state of votes? What prob­lems do you want to solve on LW that votes or some­thing else may be solu­tions to? And how would you im­prove votes or some­thing else to solve those prob­lems?