2018 Review: Voting Results!

The votes are in!

59 of the 430 el­i­gible vot­ers par­ti­ci­pated, eval­u­at­ing 75 posts. Mean­while, 39 users sub­mit­ted a to­tal of 120 re­views, with most posts get­ting at least one re­view.

Thanks a ton to ev­ery­one who put in time to think about the posts—nom­i­na­tors, re­view­ers and vot­ers al­ike. Sev­eral re­views sub­stan­tially changed my mind about many top­ics and ideas, and I was quite grate­ful for the au­thors par­ti­ci­pat­ing in the pro­cess. I’ll men­tion Zack_M_Davis, Vanessa Kosoy, and Daniel Filan as great peo­ple who wrote the most up­voted re­views.

In the com­ing months, the LessWrong team will write fur­ther analy­ses of the vote data, and use the in­for­ma­tion to form a se­quence and a book of the best writ­ing on LessWrong from 2018.

Below are the re­sults of the vote, fol­lowed by a dis­cus­sion of how re­li­able the re­sult is and plans for the fu­ture.

Top 15 posts

  1. Embed­ded Agents by Abram Dem­ski and Scott Garrabrant

  2. The Rocket Align­ment Prob­lem by Eliezer Yudkowsky

  3. Lo­cal Val­idity as a Key to San­ity and Civ­i­liza­tion by Eliezer Yudkowsky

  4. Ar­gu­ments about fast take­off by Paul Christiano

  5. The Costly Co­or­di­na­tion Mechanism of Com­mon Knowl­edge by Ben Pace

  6. Toward a New Tech­ni­cal Ex­pla­na­tion of Tech­ni­cal Ex­pla­na­tion by Abram Demski

  7. Anti-so­cial Pu­n­ish­ment by Martin Sustrik

  8. The Tails Com­ing Apart As Me­taphor For Life by Scott Alexander

  9. Bab­ble by alkjash

  10. The Loud­est Alarm Is Prob­a­bly False by orthonormal

  11. The In­tel­li­gent So­cial Web by Valentine

  12. Pre­dic­tion Mar­kets: When Do They Work? by Zvi

  13. Co­her­ence ar­gu­ments do not im­ply goal-di­rected be­hav­ior by Ro­hin Shah

  14. Is Science Slow­ing Down? by Scott Alexander

  15. A vot­ing the­ory primer for ra­tio­nal­ists by Jame­son Quinn and Ro­bust­ness to Scale by Scott Garrabrant

Top 15 posts not about AI

  1. Lo­cal Val­idity as a Key to San­ity and Civ­i­liza­tion by Eliezer Yudkowsky

  2. The Costly Co­or­di­na­tion Mechanism of Com­mon Knowl­edge by Ben Pace

  3. Anti-so­cial Pu­n­ish­ment by Martin Sustrik

  4. The Tails Com­ing Apart As Me­taphor For Life by Scott Alexander

  5. Bab­ble by alkjash

  6. The Loud­est Alarm Is Prob­a­bly False by Orthonormal

  7. The In­tel­li­gent So­cial Web by Valentine

  8. Pre­dic­tion Mar­kets: When Do They Work? by Zvi

  9. Is Science Slow­ing Down? by Scott Alexander

  10. A vot­ing the­ory primer for ra­tio­nal­ists by Jame­son Quinn

  11. Toolbox-think­ing and Law-think­ing by Eliezer Yudkowsky

  12. A Sketch of Good Com­mu­ni­ca­tion by Ben Pace

  13. A LessWrong Crypto Au­topsy by Scott Alexander

  14. Un­rol­ling so­cial metacog­ni­tion: Three lev­els of meta are not enough. by Academian

  15. Va­ri­eties Of Ar­gu­men­ta­tive Ex­pe­rience by Scott Alexander

Top 10 posts about AI

(The vote in­cluded 20 posts about AI.)

  1. Embed­ded Agents by Abram Dem­ski and Scott Garrabrant

  2. The Rocket Align­ment Prob­lem by Eliezer Yudkowsky

  3. Ar­gu­ments about fast take­off by Paul Christiano

  4. Toward a New Tech­ni­cal Ex­pla­na­tion of Tech­ni­cal Ex­pla­na­tion by Abram Demski

  5. Co­her­ence ar­gu­ments do not im­ply goal-di­rected be­hav­ior by Ro­hin Shah

  6. Ro­bust­ness to Scale by Scott Garrabrant

  7. Paul’s re­search agenda FAQ by zhukeepa

  8. An Un­trol­lable Math­e­mat­i­cian Illus­trated by Abram Demski

  9. Speci­fi­ca­tion gam­ing ex­am­ples in AI by Vika

  10. 2018 AI Align­ment Liter­a­ture Re­view and Char­ity Com­par­i­son by Larks

The Com­plete Results

Click Here If You Would Like A More Com­pre­hen­sive Vote Data Spreadsheet

To help users see the spread of the vote data, we’ve in­cluded swarm­plot vi­su­al­iza­tions.

  • For space rea­sons, only votes with weights be­tween −10 and 16 are plot­ted. This cov­ers 99.4% of votes.

  • Gridlines are spaced 2 points apart.

  • Con­crete illus­tra­tion: The plot im­me­di­ately be­low has 18 votes rang­ing in strength from −3 to 12.

#Post Ti­tle To­tal Vote Spread
1Embed­ded Agents209
(One out­lier vote of +17 is not shown)
2The Rocket Align­ment Prob­lem183
3Lo­cal Val­idity as a Key to San­ity and Civ­i­liza­tion133
4Ar­gu­ments about fast take­off98
5The Costly Co­or­di­na­tion Mechanism of Com­mon Knowl­edge95
6Toward a New Tech­ni­cal Ex­pla­na­tion of Tech­ni­cal Ex­pla­na­tion91
7Anti-so­cial Pu­n­ish­ment90
(One out­lier vote of +20 is not shown)
8The Tails Com­ing Apart As Me­taphor For Life89
9Bab­ble85
10The Loud­est Alarm Is
Prob­a­bly False
84
11The In­tel­li­gent So­cial Web79
12Pre­dic­tion Mar­kets:
When Do They Work?
77
13Co­her­ence ar­gu­ments do not im­ply goal-di­rected be­hav­ior76
14Is Science Slow­ing Down?75
15Ro­bust­ness to Scale74
15A vot­ing the­ory primer for ra­tio­nal­ists74
17Toolbox-think­ing and Law-think­ing73
18A Sketch of Good Com­mu­ni­ca­tion72
19A LessWrong Crypto Au­topsy71
20Paul’s re­search agenda FAQ70
21Un­rol­ling so­cial metacog­ni­tion: Three lev­els of meta are not enough.69
22An Un­trol­lable Math­e­mat­i­cian Illus­trated65
23Speci­fi­ca­tion gam­ing
ex­am­ples in AI
64
23Will AI See Sud­den Progress?64
23Va­ri­eties Of Ar­gu­men­ta­tive Ex­pe­rience64
26Meta-Hon­esty: Firm­ing Up Hon­esty Around Its Edge-Cases62
27My at­tempt to ex­plain
Look­ing, in­sight med­i­ta­tion,
and en­light­en­ment in
non-mys­te­ri­ous terms
60
27Nam­ing the Name­less60
27Inad­e­quate Equil­ibria vs. Gover­nance of the Com­mons60
302018 AI Align­ment Liter­a­ture Re­view and Char­ity Com­par­i­son57
31Notic­ing the Taste of Lo­tus55
31On Do­ing the Im­prob­a­ble55
31The Pavlov Strat­egy55
31Be­ing a Ro­bust, Co­her­ent Agent (V2)55
35Spaghetti Tow­ers54
36Beyond Astro­nom­i­cal Waste51
36Re­search: Res­cuers dur­ing the Holo­caust51
38Open ques­tion: are min­i­mal cir­cuits dae­mon-free?48
38De­cou­pling vs Con­tex­tu­al­is­ing Norms48
(One out­lier vote of +23)
40On the Loss and Preser­va­tion of Knowl­edge47
41Is Click­bait De­stroy­ing Our Gen­eral In­tel­li­gence?46
42What makes peo­ple in­tel­lec­tu­ally ac­tive?43
43Why ev­ery­thing might have taken so long40
44Challenges to Chris­ti­ano’s ca­pa­bil­ity am­plifi­ca­tion pro­posal39
45Public Po­si­tions and Pri­vate Guts38
46Clar­ify­ing “AI Align­ment”36
46Ex­pres­sive Vo­cab­u­lary36
48Bot­tle Caps Aren’t Op­ti­misers34
49Ar­gue Poli­tics* With Your Best Friends32
50Player vs. Char­ac­ter: A Two-Level Model of Ethics30
51Con­ver­sa­tional Cul­tures: Com­bat vs Nur­ture (V2)29
51Act of Char­ity29
53Op­ti­miza­tion Am­plifies27
53Cir­cling27
(One out­lier vote of −17)
55Real­ism about ra­tio­nal­ity25
(Two out­liers of −30 and +18)
55Car­ing less25
57Les­sons from the Cold War on In­for­ma­tion Hazards: Why In­ter­nal Com­mu­ni­ca­tion is Crit­i­cal24
57The Bat and Ball Prob­lem Re­vis­ited24
59Ar­gu­ment, in­tu­ition, and re­cur­sion21
59Un­known Knowns21
61Com­pet­i­tive Mar­kets as Distributed Back­prop18
62Towards a New Im­pact Mea­sure14
62Ex­plicit and Im­plicit Com­mu­ni­ca­tion14
62On the Chatham House Rule14
62His­tor­i­cal math­e­mat­i­ci­ans ex­hibit a birth or­der effect too14
66Every­thing I ever needed to know, I learned from World of War­craft: Good­hart’s law13
67The fun­nel of hu­man ex­pe­rience11
68Un­der­stand­ing is trans­la­tion9
69Pre­limi­nary thoughts on
moral weight
7
70Me­taphilo­soph­i­cal com­pe­tence can’t be dis­en­tan­gled from al­ign­ment3
71Two types of math­e­mat­i­cian2
72How did academia en­sure pa­pers were cor­rect in the early 20th Cen­tury?-2
73Birth or­der effect found in No­bel Lau­re­ates in Physics-5
74Give praise-10
75Affor­dance Widths-142
(One out­lier of −29)

How re­li­able is the out­put of this vote?

For most posts, be­tween 10-20 peo­ple voted on them (me­dian of 17). A change by 10-15 in a post’s score is enough to move a post up or down around 10 po­si­tions within the rank­ings. This is equal to a few mod­er­ate strength votes from two or three peo­ple, or an ex­ceed­ingly strong vote from a sin­gle strongly-feel­ing voter. This means that the sys­tem is some­what noisy, though it seems to me very un­likely that posts at the very top could end up placed much differ­ently.

The vote was also af­fected by two tech­ni­cal mis­takes the team made:

  1. The post-or­der was not ran­dom­ized. For the first half of the vot­ing pe­riod, the posts on the vot­ing page ap­peared in or­der of num­ber of nom­i­na­tions (least to most) in­stead of ap­pear­ing ran­domly, thereby giv­ing more vi­sual at­ten­tion to the first ~15 or so posts (these were posts with 2 nom­i­na­tions). Ruby looked into it and says that 15-30% more peo­ple cast votes on these ear­lier-ap­pear­ing posts com­pared to those ap­pear­ing el­se­where in the list. Thanks to gjm for iden­ti­fy­ing this is­sue.

  2. Users were given some free nega­tive votes. When calcu­lat­ing the cost of users’ votes, we used a sim­ple equa­tion, but missed that it pro­duced an off-by-one er­ror for nega­tive num­bers. Essen­tially, users got a free 1-nega­tive-vote-weight on all the posts to which they had voted on nega­tively. To cor­rect for this, for those who had ex­ceeded their bud­get − 18 users in to­tal—we re­duced the strength of their nega­tive votes by a sin­gle unit, and for those who had not spent all their points their votes were un­af­fected. This didn’t af­fect the rank-or­der­ing very much, a few posts changed by 1 po­si­tion, and a smaller num­ber changed by 2-3 po­si­tions.

The effect size of these er­rors is not cer­tain since it’s hard to know how peo­ple would have voted coun­ter­fac­tu­ally. My sense is that the effect is pretty small, and that the ma­jor­ity of noise in the sys­tem comes from el­se­where.

Fi­nally, we dis­carded ex­actly one bal­lot, which spent 10,000 points on vot­ing in­stead of the al­lot­ted 500. Had a user gone over by a small amount e.g. 1-50 points, we had planned to scale their votes down to fit the bud­get. How­ever when some­one’s al­lo­ca­tion was so ex­treme, we were hon­estly un­sure what ad­just­ment to their votes they would have wanted, as if their points had been nor­mal­ised down to 500, the ma­jor­ity of their votes would have been ad­justed to zero. (This de­ci­sion was made with­out know­ing the user who cast the bal­lot or which posts were af­fected.)

Over­all, I think the vote is a good in­di­ca­tor to about 10 places within the rank­ings, but, for ex­am­ple, I wouldn’t ag­o­nise over whether a post is at po­si­tion #42 vs #43.

The Future

This has been the first LessWrong An­nual Re­view. This pro­ject was started with the vi­sion of cre­at­ing a piece of in­fras­truc­ture that would:

  1. Create com­mon knowl­edge about how the LessWrong com­mu­nity feels about var­i­ous posts and top­ics and the progress we’ve made.

  2. Im­prove our longterm in­cen­tives, feed­back, and re­wards for au­thors.

  3. Help cre­ate a highly cu­rated “Best of 2018” Se­quence and Book.

The vote re­veals much dis­agree­ment be­tween LessWrongers. Every post has at least five pos­i­tive votes and ev­ery post had at least one nega­tive vote – ex­cept for An Un­trol­lable Math­e­mat­i­cian Illus­trated by Abram Dem­ski, which was ev­i­dently just too like­able – and many peo­ple had strongly differ­ent feel­ings about many posts. Many of these seem more in­ter­est­ing to me than the spe­cific rank­ing of the given post.

In to­tal, users wrote 207 nom­i­na­tions and 120 re­views, and many au­thors up­dated their posts with new think­ing, or clearer ex­pla­na­tions, show­ing that both read­ers and au­thors re­flected a lot (and I think changed their mind a lot) dur­ing the re­view pe­riod. I think all of this is great, and like the idea of us hav­ing a Schel­ling time in the year for this sort of think­ing.

Speak­ing for my­self, this has been a fas­ci­nat­ing and suc­cess­ful ex­per­i­ment—I’ve learned a lot. My thanks to Ray for push­ing me and the rest of the team to ac­tu­ally do it this year, in a move-fast-and-break-things kind of way. The team will be con­duct­ing a Re­view of the Re­view where we take stock of what hap­pened, dis­cuss the value and costs of the Re­view pro­cess, and think about how to make the re­view pro­cess more effec­tive and effi­cient in fu­ture years.

In the com­ing months, the LessWrong team will write fur­ther analy­ses of the vote data, award prizes to au­thors and re­view­ers, and use the vote to help de­sign a se­quence and a book of the best writ­ing on LW from 2018.

I think it’s awe­some that we can do things like this, and I was hon­estly sur­prised by the level of com­mu­nity par­ti­ci­pa­tion. Thanks to ev­ery­one who helped out in the LessWrong 2018 Re­view—ev­ery­one who nom­i­nated, re­viewed, voted and wrote the posts.