Spaced Repetition literature review prize: And the winner is...
The Spaced Repetition literature review prize for the best new review of the evidence on Spaced Repetition has ended and the judging panel has made its decision. The prize attracted entries from Duke (entry) and Gwern (entry). After reviewing the submissions separately and then discussing them together, the judging panel unanimously judged Gwern’s entry to be the best.
With great pleasure, we now award Gwern the prize of $385.
Anki cards for Gwern’s review are available as the shared deck “Gwern Spaced Repetition Lit Review”. If you have improvements or alternate decks, post them in the comments.
The three judges, BenLowell, Guy Srinivasan and John Salvatier (me), are Seattle LessWrongers who volunteered to judge the contest.
We thank Duke and Gwern for their submissions, as well as users randomwalker, Antisuji, Dr_Manhattan, Benquo, Nick_Roy for contributing to the prize fund.
- [prize] new contest for Spaced Repetition literature review ($365+) by (18 Jun 2011 18:31 UTC; 23 points)
- What are good topics for literature review prizes? by (2 Sep 2011 23:48 UTC; 7 points)
- 's comment on LessWrong as place for scientifically literate advice by (20 Aug 2011 0:28 UTC; 2 points)
- 's comment on How can we get more and better LW contrarians? by (20 Apr 2012 23:48 UTC; 2 points)
- 's comment on Less Wrong Product & Service Recommendations by (6 Jul 2012 9:40 UTC; 1 point)
With all that said, I do have some feedback for jsalvatier! (Please don’t take this as defecting criticism; also, in the interests of honesty—I noticed all of #2 shortly after the contest began, and kept quiet out of plain self-interest, waiting until the judging happened).
First, it caused me a little stress that judging took so long. The deadline was 1 August, it didn’t have to take 18 days to judge, even if events intervened (as they apparently did) to block the earlier gathering which would judge the contest—it could have been done online by so few judges. This is especially true for a contest with only two entries.
The contest was mishandled from the standpoint of publicity. I should not have been so confident I would win; there should have been more than 2 entries with $400 at stake for a task intrinsically worthwhile. There are a number of ways this could have been advertised better:
This could have been announced on the Mnemosyne and Anki mailing lists and forums. It is quite relevant to their interests and they are the obvious places to announce such a contest, even if they would not link it on their front pages.
It could have been advertised better on LW. This was really generous and interesting and potentially a key community tool! (lukeprog won’t be this active forever.) Why wasn’t the contest an article on the Front Page? (AFAIK, it never appeared there.)
Piotr Wozniak, the expert of experts on spaced repetition in practice, could have been contacted to participate or possibly advertise it on Supermemo.com; he may sound intimidating from the Wired article, but my email correspondence with him was perfectly normal and fluid, with the ordinary delays.
The contest, per #2, could have been submitted to Reddit (the psychology subreddit being only the most obvious of multiple potential targets) or Hacker News; both communities are interested in spaced repetition. This is also true of Imminst.
Google AdWords could have helped; $10 buys a lot of clicks when it comes to spaced-repetition ads, and could well be worth it. ($350 is about as impressive as $360 as seed for a prize.)
Interested LessWrongers could have been personally contacted. Anyone who has created a deck of flashcards is obvious; and why not contact lukeprog? Reading and summarizing papers and reviews is practically his specialty. He is busy, of course, but it’s not like spaced repetition is that worthless compared to his current topics, and the prize may make up for lost time.
1 I did actually announce this on the Mnemnosyne and Anki mailing lists, but I received no responses. :(
2 I felt timid about advertising this on the mainpage, though I am not sure why in retrospect.
3-5 are good suggestions I hadn’t thought of. The question is how to generate similar suggestions for prizes in the future. I suppose 5 is obviously applicable, 4 probably applies to many contests, I’m not sure about 3. Perhaps ask if there are any famous figures in the field I am trying to summarize.
6 feels like it wouldn’t be effective to me, but I’m not sure why. I suppose the costs to trying this in the future are low.
Thanks again :)
Whups! So you’re right. I replied with news about the verdict.
#2 is definitely something you should try. If there were no real prize, I think people might laugh at an attempt for the main page, but with that much skin in the game… at least, I would expect LessWrongers to respect someone putting a good chunk of money where their mouth is. They seem to’ve in the past, with reports about bets, prediction market trades, and SIAI donations all getting upvoted.
I suspect you feel about #6 the same way you feel about #2 - that it’s overreaching and arrogant. That is something you may want to look into if you feel the same way in a second contest of equivalent quality. (I understand ‘rejection therapy’ may help with that.)
EDIT: I’ve spent a fair bit of time messing with AdWords for my own purposes, and I think I have a handle on the basics, so I’d be willing to help out with future contests as far as that goes.
I think I would feel much more confident about a second prize. In the beginning I didn’t have other people helping me do this kind of work, but now I do.
I agree with all of this, and thanks for the feedback. For my part in causing the judging to take long I apologize; I didn’t try to delay it, but I definitely didn’t put any explicit effort into speeding it up either. Publicity is one of the things we discussed in our brief retrospective. It’s likely another contest will take place, and if it does, I will make an effort to spread the word, but am not committing to make such an effort that no more will be needed. ;)
Thanks gwern!
I’d like to thank John for putting up the prize money and just starting the prize—such things can be thankless, and as public goods one might expect them to be under-supplied. I would have worked in some of that research, but not this soon nor in as much depth.
I also appreciate the feedback he and the judges left on the SR page. Feedback is a pretty rare thing for things I write.
I’m afraid I don’t have grandiose plans for the money - $272 is earmarked for repairs to my car (in part thanks to my sister), and I already spent $70 on fish oil and vitamin D. The $40 left may enable me to finally experiment with nicotine, which ought to be interesting.
EDIT: I ordered the nicotine-water last night. Preliminary plans: using the habit-formation for exercise, spaced repetition, and n-backing.
Upvoted for putting your money to good use. I enjoy your work on nootropics and am looking forward to reading more from you.
Your website might have stripped out the division sign. I see:
Took me a sec to figure out it should be 100 x 1000 / 2.
That’s just you. Your web browser is not running Javascript, you don’t have the fonts, or something. I see the division as I wrote it in Firefox: http://i.imgur.com/c2U2R.png
I see the HTML source uses MathML:
I couldn’t tell from the screenshot what browser and OS you’re using but for me (on Linux), it shows up as “100×10002” in Chrome but as a (correct) fraction in Firefox (3.5). In both it acts like “100×10002″ in the contexts of searching and copying/pasting.
Ditto—on Arch linux, chrome with javascript enabled does not work; firefox 3.6 does.
Correct. The Javascript is because apparently browsers have to be forced to render the MathML; I don’t understand the details.
Debian unstable’s Iceweasel.
I tried three browsers and in none of them did javascript make a difference. (I had both success and failure.)
(Just in case it wasn’t clear, I do have JavaScript enabled in Chrome.)
I had a six month long “4-6 days a week” Anki habit under development before I started my most recent gig. The new work routines disrupted my Anki habit and several days ago, actually using some of the content, I realized I’d forgotten how to fingerspell a certain letter that was in one of my decks. Reading these essays reminded me of all the good reasons I had for doing Anki and I started working through my backlog of cards.
I just wanted to say thanks for the contest and the essays and everything. It is more likely than not that I’ll reform my habit in my new environment. You caused the splash and I benefited from the positive ripples. Thank you to everyone involved :-)
Ok. I had earlier promised to donate $20 to the charity of the winner’s choice. So, Gwern please choose.
Oh! Oh, OK then—donate to the Internet Archive, please.
(No doubt people would disagree because it comes out very badly on a utilitarian calculation, but this is more about discharging my personal emotion of obligation—I use the Archive on a literally daily basis.)
Done. Receipt number is 4518-9128-6646-8864.
Thanks.
Curious what do you find so useful other than looking up ol’ webpages
http://www.gwern.net/Archiving%20URLs
There’s also a lot of media on IA; I got a good chunk of Turkish and instrumental music there, as well as a number of movies.
People have weird utility functions.
My biggest takeaway from the research is that it’s worth thinking briefly about every new Anki card whether I can make it an effective short-answer question rather than a cloze-deletion.
Er… that’s my biggest action item. My biggest takeaway is that research definitely supports spaced repetition + testing far over studying.
I actually hadn’t noticed that Spaced Repetition is testing till this research. I didn’t know about the testing effect. I think knowing about the testing effect will make me more effective learning things more generally. I’d actually like to see an stand alone article on the testing effect.