The 2021 Review Phase
(Apologies for this post being quite late – I got sick just prior to the holidays, and then had a surprisingly eventful holidays)
We’ve wrapped up the nominations phase for the 2021 Review. 139 people have participated so far, casting 1815 votes and writing 72 reviews.
Now we’re in the Review Phase, where we ask “Were these posts actually good? How could they be improved? How do they fit together?”.
Key Info:
Posts need at least one review in order to progress to the final voting stage. To write a review, navigate to a nominated post page, and click the “Review” button at the top of the page.
I’ll award prizes to reviews that I think offer substantive information. Roughly $50 for a (See “What kinds of reviews are most valuable?” for examples)
You can look over the results of last year’s review at /reviewVoting/2020, to compare the results there to this year’s.
You can sort the posts on the /reviewVoting page by “Vote Total”, to see which posts were most upvoted during the nomination voting. Thorough reviews of the top-scoring posts are particularly valuable.
How much Progress Overall?
One of my original goals for the Review was to provide a clearer feedback mechanism to the Lightcone team on “how is LessWrong doing?”. It’s one thing to track karma and site activity. But did our infrastructure maintenance, UI improvements, research retreats and other interventions actually seem to translate into more good, quality intellectual progress?
This year some people have commented that there are fewer exciting-feeling posts than they remember from previous years. I’ve also had a bit of this feeling – despite feeling at the time like in 2021, LessWrong was a pretty happening place with lots of good content. I’m not sure I endorse the feeling. For example: I think important progress happened in the MIRI Dialogues, but in a diffuse way, which didn’t result in many concrete posts I can point at to that I learned discrete concepts from. Progress != posts, necessarily.
But, it still seems worth paying attention to the sensation. Yes requires the possibility of no, and it’s fairly important for the Review to be able to return the answer “less overall progress this year.”
Now, right now there isn’t much of a mechanism for tracking “how much progress” there was between years. I’m still thinking about how to seriously answer this question. But for now, I’ve update the /reviewVoting page to be able to show previous years, so you can go eyeball the results from the previous 2020 Review, see how many posts you voted as a “9″, etc.
What kinds of reviews are most valuable?
Briefly noting what you liked or didn’t like about a post is good. But here are some specific things I’m excited for:
New Information. My favorite reviews are ones that give new information to the reader. This could be a concrete fact, or a rebuttal to a key argument in a post, or a new way the post is relevant that you hadn’t considered. It could be a new frame about how to even think about the post.
Concrete use cases. I think reviews that say “this concretely helped me, here’s how” are also great, especially if they give specifics. I think it’s both helpful and rewarding to individual authors to hear how their post was useful, so they can do more of that. (I think this is also helpful on a broader level, so that the collective LW userbase can see what kinds of effects are realistic)
i.e. rather than saying “this post has helped me a bunch over the years”, say “here are two specific times that it helped me, and how.”
Thoughts on how the post could be improved. Two years later, if a post still seems like good reference material, but is confusing or poorly argued, give advice on how to improve it. Discuss the use-case for the readership you have in mind.
Reflect on the Big Picture. How do various posts fit together into something greater than the sum of their parts? What major conversations have happened on LessWrong and what have you taken from them?
Brevity/clarity. Bonus points if they are concise and relevant enough that it’s fairly easy to print them in the 2021 book. (I wanted to print some reviews in the 2020 books but found it’d require a fair amount of editing work to get them into a shape that made sense). Note, I’m not saying “optimize for shortness at the expense of saying anything substantive”, just, note that all-else-being-equal, taking less space to convey the key information is helpful.
Go Forth, Review, and Get Prizes
Remember, I’ll be awarding prizes to reviews that I think add substantial new information, critique, or useful frames. Prizes will be announced in the comments here.
I’ve made a few updates to the Review UI over the past week. (I apologize to people who keep having the UI shift a little bit underneath them, although I think it’s overall an improvement).
Quick Review Page
I originally designed the Review Dashboard as basically a powertool for powerusers. The Review is kinda intrinsically a complex operation, and for my own use case, I think I needed a view that showed a lot of information. I went and tried to simplify it recently, and found that, nope, all the complexity was actually serving an important purpose.
But, it sure was overwhelming. And having thought through the ideal poweruser use-case, I finally had an easier time figuring out a more simplified UI that tries to just get you to do one thing, reasonably easily.
So, behold the Quick Review Page. This basically tells you “please try to review 3 posts if you can”, and shows you the top-10 unreviewed posts (based on the votes from 1000+ karma users in the Nomination Voting phase).
It doesn’t cover everything you might possibly want. There’s a lot of posts I think would benefit from multiple reviews (Quick Review only shows un-reviewed posts). So I do recommend still checking out the Advanced Review Dashboard, which provides more customized sorting options. But, if you look at the advanced dashboard and go “aaaaah”, or “uuugggh why does it take so long to load?”, the Quick Review page loads much faster and probably gets you most of the value.
3 Review “checkboxes”
I think it’s a reasonable “be a good citizen of LessWrong” ask to do at least 3 quick reviews, to help provide some longterm signal. I’m most interested in this from longterm users so on the frontpage it only shows from 1000+ karma users, although they appear for all users on the dashboard pages.
This is just a UI nudge, obviously a lot of people are busy, or aren’t that invested in LessWrong, and that’s fine.
Reviews page
You can now easily see reviews from previous years on the /reviews page. You can filter by year, or see all reviews from all time.
Longform Reviews
Some people have historically written up longform, top-level-post reviews, that either go into a lot more detail evaluating one post, or review a lot of posts at once and maybe look at some high level themes that emerge. I wanted to more explicitly encourage people to do that. So there is now a button on the dashboard pages
You can find it at the bottom-right of the Quick Review or Advanced Dashboard. Functionally it’s only slightly different from the normal New Post edit form. In future years I might incorporate Top Level Review posts into the rest of the Review UI.
We are now on to the final voting phase for the review. I may write up a post about that later, although the main takeaway is, well “cast your final ballots.” The UI for the voting phase integrates the reviews into the voting ballots, so you can read a bit about what others found valuable or wrong/confused about each post to help inform your votes.
Meanwhile, I do want to note: you can still review as-yet-unreviewed posts, and get them onto the ballot. The Voting Dashboard won’t show posts that haven’t been reviewed, so you’ll have to remember the individual posts and navigate to their post page to find the “review” button. (This wasn’t originally intentional behavior, I realized it was possible to continue reviewing things into the Voting Phase a couple years ago, but, I decided it’s basically fine for the Voting Phase to be a “soft deadline”, and left it as a “feature” rather than a “bug”. The longer you wait to review things the fewer people will see it and it’s less likely to get upvoted, so, I do encourage you to get any last minute reviews in soon)
Also meanwhile, I still think it’s valuable to write in-depth reviews, especially for posts you expect to do well in the final vote. In my ideal world, all the top 10-20 posts would have gotten subjected to a significant critical analysis.
Meanwhile meanwhile meanwhile, a couple days ago I coded up a review leaderboard. This was kinda last minute and mostly I’m excited to have be more integrated into next year’s 2022 Review, but, feel free to get Extrinsically Motivated to do some additional reviews. :P [also remember sock puppet voting will get your account banned :p :/]. Congrats to Elizabeth, Daniel Kokotajlo, la3orn, Drake Morrison and A Ray for being the non-LessWrong-team-members who topped the chart so far.
After talking things over with the LessWrong team, I decided to re-run the preliminary-vote totals. So, during the final week of the Review Phase, people deciding what to prioritize have a more accurate picture of which posts are likely to actually win, and therefore are worth more effort writing detailed reviews of.
It’s important not to be continuously re-running the vote-totals because it shifts voting behavior to be too strategic, but I think it’s fine to re-run them once in the middle of the review phase each year.
This thread will be for aggregating reviews that I awarded prizes to.
This comment is incomplete, and I will likely edit some of the prizes slightly. But, I’d fallen behind on awarding prizes for reviews and I want to highlight Yes, the Lightcone team will give you money for reviewing stuff. So, I wanted to ship this rough version for now to give a sense of what sort of reviews I found valuable.
Thanks to the large number of people who’ve stepped up to review so far. (I’d still be excited for EffortReview that explore the details of some of the higher ranked posts)
Next round of prize announcements:
$200 to Nostalgebraist for this review of Fun with +12 OOMs of Compute. I think I didn’t really get the underlying point of the post until I read this.
$200 to Akash for his review of ARC’s first technical report: Eliciting Latent Knowledge, breaking out a lot of details about how ELK affected Akash and people in his cohort, and ways he thinks ELK could be improved.
$150 to Alex Altair’s review of Taboo Outside View.
$150 to la3orn for their self-review of EfficientZero: How It Works, in particular for grading their conclusions and predictions.
$100 to A Ray’s review of Coase’s “Nature of the Firm” on Polyamory for explaining something subtly off about
$100 to A Ray for his review of Politics is way too meta
$100 to A Ray for his review of Cryonics Signup Guide #1: Overview
$100 to Leon Lang for his fairly detailed review of Utility Maximization = Description Length Minimization.
$100 to Valentine for his self review of Creating a Truly Formidable Art.
$100 to Zach Davis for his self-review of Feature Selection, in particular for situating it more clearly in the context of his other work, and previous work it was building on.
$100 to Dragon God for his review of EfficientZero: How It Works, which dug into some details and found a potential error.
$100 to Vanessay Kosoy for their self-review of Infra-Bayesianism. I appreciated her laying out what developments had progressed in the intervening year.
$100 to Neel Nanda for his self-review of Intentionally Making Close Friends
$100 to Steven Byrnes review of Robin Hanson’s Grabby Aliens model explained
$100 to Turntrout for his self-review of Lessons I’ve Learned From Self-Teaching. I love that he had some data on how his attempts had faired.
$100 to Adam Shimi for for his review of Biology-Inspired AGI Timelines: The Trick That Never Works. I found this particularly valuable for breaking out some of the mistakes Eliezer is pointing out in a fairly distilled fashion. (I personally had trouble actually taking away the points from the post itself)
$100 to Vaniver for his review of ARC’s first technical report: Eliciting Latent Knowledge.
$100 to la3orn for their review of Coordination Schemes are Capital Investments.
$100 to Rafael Harth for their review of [Book review] Gödel, Escher, Bach: an in-depth explainer
$75 to Alex Flint for his self-review of Agency in Conway’s Game of Life. I appreciated the link to the Game of Life community where people were researching a question Alex had posted. (My impression is they were working on that independent of Alex’s work)
$100 to MondSemmel for his review of Where did the 5 micron number come from? Nowhere good?
$75 to Adam Shimi for his review of What Multipolar Failure Looks Like, and Robust Agent-Agnostic Processes.
$75 to Rafael Hart’s review of Dear Self: We Need to Talk About Social Media. I particularly was interested in the articulation of how “vibe” comes across in posts, and why that matters. I did feel a bit surprised that Rafael found the particular vibe in Elizabeth’s piece and not in many other LessWrong pieces (maybe he did, and just noticed it here?), but I found it a helpful frame for thinking about what value I get from posts.
Honorable Mentions:
These reviews didn’t go into much detail, but added at least a couple new arguments or frames that I found useful:
Zach Stein-Perlman for their review of Fun with +12 OOMs of Compute. I appreciated him laying out the specific reasoning for how it improved his thinking.
Adam Jermyn for their review of Your Cheerful Price.
gears to ascension for their review of Your Cheerful Price.
AllAmericanBreakfast for their review of Oliver Sipple. I particularly liked the evidence-evaluation investigation of “on one hand, it’d be weird to have two unrelated phenomena happening at the same time, so maybe the story just checks out. On the other hand, the two phenomena are only promotedto our attention because of
Daniel Koko for his self-review of Birds, Brains, Planes and AI.
Darmani for their self-review of Leaky Delegation: You Are Not A Commodity, with details about which of their attempts at employing their own advice hadn’t worked out
Eli Tyre for his self-review of How do we prepare for the final crunch time?
John Wentworth on Highlights from the Autobiography of Andrew Carnegie
Double’s review of Against Neutrality About Creating Happy Lives.
Alex Altair for his review of Lessons I’ve Learned From Self Teaching.
Valentine for his review of Gravity Turn
Daniel Kokotajlo for his review of Comments on Carlsmith’s “Is power-seeking AI an existential risk?”
Daniel Kokotaljo for his review of Fixing The Good Regulator Theorem
A Ray for his review of Rationalism Before the Sequences. Reading the review got me thinking more about the value of good bibiographies.
Coafos for his review of Social Behavior Curves, Equilibria and Radicalism
Coafos for his review of Curing Insanity with Malaria.
Srdjan Miletic for his review of https://www.lesswrong.com/posts/gziZACDg6EBpGZbJe/almost-everyone-should-be-less-afraid-of-lawsuits?commentId=i6B94ofbtKBdKridn
Akash for his review of How To Get Into Independent Research On Alignment/Agency
Next set of review prizes:
$150 to Yoav Ravid for their review of What do GDP Growth Curves Really Mean? I really liked that they did a mini-distillation of the older good comments about the post.
$100 to Leon Lang for their review of the Telephone Theorem.
$50 to Unnamed for for their review of Taboo Outside View
$50 to Srdjan Miletic for their review of Almost everyone should be less afraid of lawsuits
First set of review prizes:
$50 to iceman’s review of Unnatural Categories are Optimized for Deception. I didn’t feel like I learned new things from it exactly, but I do appreciate iceman laying out his reasoning, what he found valuable, and some specific reasoning that he may have done better because of it.
$100 to Logan Strohl for their self-review of Catching the Spark. I like that Logan brought a lot more experience to bear on an earlier topic, including noticing a place where their original frame was maybe 50% wrong.
$100 to Daniel Kokotajlo for his self review of What 2026 looks like.
$200 to AllAmericanBreakfast for both his review and followup comment on Core Pathways of Aging.
If he were eligible for a prize, I’d be awarding $50 to Ben Pace’s review of Yudkowsky and Christiano discussing takeoff speeds. (Like iceman’s review of Unnatural Categories, I didn’t quite feel like I learned from it per se, but still was glad to get a detailed breakdown of what Ben found valuable).
If you want to receive money, please update your payment contact info and PayPal email at https://www.lesswrong.com/payments/account