The LessWrong 2018 Review

LessWrong is cur­rently do­ing a ma­jor re­view of 2018 — look­ing back at old posts and con­sid­er­ing which of them have stood the tests of time. It has three phases:

  • Nom­i­na­tion (ends Dec 1st at 11:59pm PST)

  • Re­view (ends Dec 31st)

  • Vot­ing on the best posts (ends Jan­uary 7th)

Authors will have a chance to edit posts in re­sponse to feed­back, and then the mod­er­a­tion team will com­pile the best posts into a phys­i­cal book and LessWrong se­quence, with $2000 in prizes given out to the top 3-5 posts and up to $2000 given out to peo­ple who write the best re­views.

Helpful Links:


This is the first week of the LessWrong 2018 Re­view – an ex­per­i­ment in im­prov­ing the LessWrong Com­mu­nity’s longterm feed­back and re­ward cy­cle.

This post be­gins by ex­plor­ing the mo­ti­va­tions for this pro­ject (first at a high level of ab­strac­tion, then get­ting into some more con­crete goals), be­fore div­ing into the de­tails of the pro­cess.

Im­prov­ing the Idea Pipeline

In his LW 2.0 Strate­gic Overview, habryka noted:

We need to build on each other’s in­tel­lec­tual con­tri­bu­tions, archive im­por­tant con­tent, and avoid pri­mar­ily be­ing news-driven.

We need to im­prove the sig­nal-to-noise ra­tio for the av­er­age reader, and only broad­cast the most im­por­tant writing

[...]

Modern sci­ence is plagued by se­vere prob­lems, but of hu­man­ity’s in­sti­tu­tions it has per­haps the strongest record of be­ing able to build suc­cess­fully on its pre­vi­ous ideas.

The physics com­mu­nity has this sys­tem where the new ideas get put into jour­nals, and then even­tu­ally if they’re im­por­tant, and true, they get turned into text­books, which are then read by the up­com­ing gen­er­a­tion of physi­cists, who then write new pa­pers based on the find­ings in the text­books. All good sci­en­tific fields have good text­books, and your un­der­grad years are largely spent read­ing them.

Over the past cou­ple years, much of my fo­cus has been on the early-stages of LessWrong’s idea pipeline – cre­at­ing af­for­dance for off-the-cuff con­ver­sa­tion, brain­storm­ing, and ex­plo­ra­tion of paradigms that are still un­der de­vel­op­ment (with fea­tures like short­form and mod­er­a­tion tools).

But, the be­gin­ning of the idea-pipeline is, well, not the end.

I’ve writ­ten a cou­ple times about what the later stages of the idea-pipeline might look like. My best guess is still some­thing like this:

I want LessWrong to en­courage ex­tremely high qual­ity in­tel­lec­tual la­bor. I think the best way to go about this is through es­ca­lat­ing pos­i­tive re­wards, rather than strong ini­tial filters.

Right now our high­est re­ward is get­ting into the cu­rated sec­tion, which… just isn’t ac­tu­ally that high a bar. We only cu­rate posts if we think they are mak­ing a good point. But if we set the cu­rated bar at “ex­tremely well writ­ten and ex­tremely epistem­i­cally rigor­ous and ex­tremely use­ful”, we would ba­si­cally never be able to cu­rate any­thing.

My cur­rent guess is that there should be a “higher than cu­rated” level, and that the gen­eral ex­pec­ta­tion should be that posts should only be put in that sec­tion af­ter get­ting re­viewed, scru­ti­nized, and most likely rewrit­ten at least once.

I still have a lot of un­cer­tainty about the right way to go about a re­view pro­cess, and var­i­ous mem­bers of the LW team have some­what differ­ent takes on it.

I’ve heard lots of com­plaints about main­stream sci­ence peer re­view: that re­view­ing is of­ten a thankless task; the qual­ity of re­view varies dra­mat­i­cally, and is of­ten en­tan­gled with weird poli­ti­cal games.

Mean­while: LessWrong posts cover a va­ri­ety of top­ics – some em­piri­cal, some philo­soph­i­cal. In many cases it’s hard to di­rectly eval­u­ate their truth or use­ful­ness. LessWrong team mem­bers had differ­ing opinions on what sort of eval­u­a­tion is most use­ful or prac­ti­cal.

I’m not sure if the best pro­cess is more open/​pub­lic (har­ness­ing the wis­dom of crowds) or pri­vate (rely­ing on the judg­ment of a small num­ber of thinkers). The cur­rent ap­proach in­volves a mix of both.

What I’m most con­fi­dent in is that the re­view should fo­cus on older posts.

New posts of­ten feel ex­cit­ing, but a year later, look­ing back, you can ask if it ac­tu­ally has be­come a helpful in­tel­lec­tual tool. (I’m also ex­cited for the idea that, in fu­ture years, the pro­cess could also in­clude re­con­sid­er­ing pre­vi­ously-re­viewed posts, if there’s been some­thing like a “repli­ca­tion crisis” in the in­ter­ven­ing time)

Re­gard­less, I con­sider the LessWrong Re­view pro­cess to be an ex­per­i­ment, which will likely evolve in the com­ing years.


Goals

Be­fore delv­ing into the pro­cess, I wanted to go over the high level goals for the pro­ject:

1. Im­prove our longterm in­cen­tives, feed­back, and re­wards for authors

2. Create a highly cu­rated “Best of 2018” se­quence /​ phys­i­cal book

3. Create com­mon knowl­edge about the LW com­mu­nity’s col­lec­tive epistemic state re­gard­ing con­tro­ver­sial posts

Longterm in­cen­tives, feed­back and rewards

Right now, au­thors on LessWrong are re­warded es­sen­tially by com­ments, vot­ing, and other peo­ple cit­ing their work. This is fine, as things go, but has a few is­sues:

  • Some kinds of posts are quite valuable, but don’t get many com­ments (and these dis­pro­por­tionately tend to be posts that are more proac­tively rigor­ous, be­cause there’s less to cri­tique, or cri­tiquing re­quires more effort, or build­ing off the ideas re­quires more do­main ex­per­tise)

  • By con­trast, com­ments and vot­ing both nudge peo­ple to­wards posts that are click­baity and con­tro­ver­sial.

  • Once posts have slipped off the front­page, they of­ten fade from con­scious­ness. I’m ex­cited for a LessWrong that re­wards Long Con­tent, that stand the tests of time, as is up­dated as new in­for­ma­tion comes to light. (In some cases this may in­volve edit­ing the origi­nal post. But if you pre­fer old posts to serve as a time-cap­sule of your post be­liefs, adding a link to a newer post would also work)

  • Many good posts be­gin with an “epistemic sta­tus: think­ing out loud”, be­cause, at the time, they were just think­ing out loud. Nonethe­less, they turn out to be quite good. Early-stage brain­storm­ing is good, but if 2 years later the early-stage-brain­storm­ing has be­come the best refer­ence on a sub­ject, au­thors should be en­couraged to change that epistemic sta­tus and clean up the post for the benefit of fu­ture read­ers.

The aim of the Re­view is to ad­dress those con­cerns by:

  • Pro­mot­ing old, vet­ted con­tent di­rectly on the site.

  • Award­ing prizes not only to au­thors, but to re­view­ers. It seems im­por­tant to di­rectly re­ward high-effort re­views that thought­fully ex­plore both how the post could be im­proved, and how it fits into the broader in­tel­lec­tual ecosys­tem. (At the same time, not hav­ing this be the fi­nal stage in the pro­cess, since build­ing an in­tel­lec­tual ed­ifice re­quires four lay­ers of on­go­ing con­ver­sa­tion)

  • Com­piling the re­sults into a phys­i­cal book. I find there’s some­thing… liter­ally weighty about hav­ing your work in printed form. And be­cause it’s much harder to edit books than blog­posts, the print­ing gives au­thors an ex­tra in­cen­tive to clean up their past work or im­prove the ped­a­gogy.

A highly cu­rated “Best of 2018” se­quence /​ book

Many users don’t par­ti­ci­pate in the day-to-day dis­cus­sion on LessWrong, but want to eas­ily find the best con­tent.

To those users, a “Best Of” se­quence that in­cludes not only posts that seemed ex­cit­ing at the time, but dis­til­led re­views and fol­lowup, seems like a good value propo­si­tion. And mean­while, helps move the site away from be­ing time-sen­si­tive-news­feed.

Com­mon knowl­edge about the LW com­mu­nity’s col­lec­tive epistemic state re­gard­ing con­tro­ver­sial posts

Some posts are highly up­voted be­cause ev­ery­one agrees they’re true and im­por­tant. Other posts are up­voted be­cause they’re more like ex­cit­ing hy­pothe­ses. There’s a lot of dis­agree­ment about which claims are ac­tu­ally true, but that dis­agree­ment is crudely mea­sured in com­ments from a vo­cal minor­ity.

The end of the re­view pro­cess in­cludes a straight­for­ward vote on which posts seem (in ret­ro­spect), use­ful, and which seem “epistem­i­cally sound”. This is not the end of the con­ver­sa­tion about which posts are mak­ing true claims that carve re­al­ity at it’s joints, but my hope is for it to ground that dis­cus­sion in a clearer group-epistemic state.


Re­view Process

Nom­i­na­tion Phase

1 week (Nov 20th – Dec 1st)

  • Users with 1000+ karma can nom­i­nate posts from 2018, de­scribing how they found the post use­ful over the longterm.

  • The nom­i­na­tion but­ton is in the post drop­down-menu (available at the top of posts, or to the right of their post-item)

  • For con­ve­nience, you can re­view posts via:

Re­view Phase

4 weeks (Dec 1st – Dec 31st)

  • Authors of nom­i­nated posts can opt-out of the re­view pro­cess if they want.

    • They also can opt-in, while not­ing that they prob­a­bly won’t have time to up­date their posts in re­sponse to cri­tique. (This may re­duce the chances of their posts be­ing fea­tured as promi­nently in the Best of 2018 book)

  • Posts with suffi­cient* nom­i­na­tions are an­nounced as con­tenders.

    • We’re aiming to have 50-100 con­tenders, and the nom­i­na­tion thresh­old will be set to what­ever gets clos­est to that range

  • For a month, peo­ple are en­couraged to look at them thought­fully, writ­ing com­ments (or posts) that dis­cuss:

    • How has this post been use­ful?

    • How does it con­nect to the broader in­tel­lec­tual land­scape?

    • Is this post epistem­i­cally sound?

    • How could it be im­proved?

    • What fur­ther work would you like to see peo­ple do with the con­tent of this post?

  • A good frame of refer­ence for the re­views are shorter ver­sions of LessWrong or Slat­es­tarCodex book re­views (which do a com­bi­na­tion of epistemic spot checks, sum­ma­riz­ing, and con­tex­tu­al­iz­ing)

  • Authors are en­couraged to en­gage with re­views:

    • Not­ing where they disagree

    • Dis­cussing what sort of fol­lowup work they’d be in­ter­ested in see­ing from others

    • Ideally, up­dat­ing the post in re­sponse to cri­tique they agree with

Vot­ing Phase

1 Week (Jan 1st – Jan 7th)

Posts that got at least one re­view pro­ceed to the vot­ing phase. The de­tails of this are still be­ing fleshed out, but the cur­rent plan is:

  • Users with 1000+ karma rate each post on a 1-10 scale, with 6+ mean­ing “I’d be happy to see this in­cluded in the ‘best of 2018’” roundup, and 10 means “this is the best I can imag­ine”

  • Users are en­couraged to (op­tion­ally) share the rea­sons for each rat­ing, and/​or share thoughts on their over­all judg­ment pro­cess.

Books and Rewards

Public Wri­teup /​ Aggregation

Soon af­ter­wards (hope­fully within a week), the votes will all be pub­li­cly available. A few differ­ent ag­gre­gate statis­tics will be available, in­clud­ing the raw av­er­age, and po­ten­tially some at­tempt at a “karma-weighted av­er­age.”

Best of 2018 Book /​ Sequence

Some­time later, the LessWrong mod­er­a­tion team will put to­gether a phys­i­cal book, (and on­line se­quence), of the best posts and most valuable re­views.

This will in­volve a lot of ed­i­tor dis­cre­tion – the team will es­sen­tially take the pub­lic re­view pro­cess and use it as in­put for the con­struc­tion of a book and se­quence.

I have a lot of un­cer­tainty about the shape of the book. I’m guess­ing it’d in­clude any­where from 10-50 posts, along with par­tic­u­larly good re­views of those posts, and some ad­di­tional com­men­tary from the LW team.

Note: This may in­volve some cus­tom edit­ing to han­dle things like hy­per­links, which may work differ­ently in printed me­dia than on­line blog­posts. This will in­volve some back-and-forth with the au­thors.

Prizes

  • Every­one whose work is fea­tured in the book will re­ceive a copy of it.

  • There will be $2000 in prizes di­vided among the au­thors of the top 3-5 posts (judged by the mod­er­a­tion team)

  • There will be up to $2000 in prizes for the best 0-10 re­views that get in­cluded in the book. (The dis­tri­bu­tion of this will de­pend a bit on what re­views we get and how good they are)

  • (note: LessWrong team mem­bers may be par­ti­ci­pat­ing as re­view­ers and po­ten­tially au­thors, but will not be el­i­gible for any awards)