[prize] Spaced Repetition literature review
EDIT: I am canceling this contest because I feel that the structure of the incentives were poorly thought out (see gwern’s comments). I will be posting a new better structured contest in the near future (now posted). If you feel this is unfair or otherwise feel slighted, please contact me at my username at gmail.com.
I’m interested in making projects happen on Less Wrong. In order to find out what works and to inspire others to try things too, I’m sponsoring the following small project:
I hereby offer a prize to the first person to submit a good summary of the evidence on Spaced Repetition to the main page. The winner will get the prize, currently: $265 + 40 to charity (see comments)
The summary should address at least the following questions:
What spacing is best?
How much does spaced repetition actually help memory?
Does spaced repetition have hidden benefits or costs?
Does the effectiveness vary across domains? How much?
Is there research on the kinds of questions that work best?
What questions do researchers think are most important?
Is there any interesting ongoing research? If so, what is it on?
What, if any, questions do researchers think it is important to answer? Are there other unanswered questions that would jump out at a smart person?
What does spaced repetition not do that people might expect it to?
Would probably help to quantify that somewhat—as in “evidence that spaced repetition does X” (X being what you want to get from it).
With this framework evidence that SRS does not do X should also get the prize.
Putting it as “claims are made that SRS does X, here is the pro/con evidence” would also be a good format.
As I also consider this topic as very valuable I’ll follow Joshua’s suggestion and offer $20 to SIAI for this post.
I’d like to work on this (be paid to substantially improve my http://www.gwern.net/Mnemosyne ? Where’s the downside?), but you ask hard questions which would require a lot of research to be done to my satisfaction and during which I could be scooped. I’m not sure a race is the best way to run this.
Why do you care if it is done to your satisfaction when the prize is awarded based on other’s satisfaction with it?
This is an expected value problem. Decide how much a unit of your time is worth, how much time you are willing to invest and then (the hard part) estimate your likelihood for success.
So, if you value your time at $10/hr, are willing to invest 10 hours and estimate you will win the $155 X% of the time then we get this equation:
P(winning)$amount won—P(losing)$amount not won = initial investment
x$155 - (1-x)$0 = $10/hr * 10 hours
solve for x
x = 64.5%, in other words, to “break even” you need to be sure that if you invest 10 hours of your time in this project that you will win it at least 64.5% of the time.
A race may not be the best way to run this for you, since I suspect that you value your time a high rate relative to the potential payoff. But someone who values their time less (or is more productive than you per unit time) may think a race is a wonderful idea.
Why do I care… what I care? Seriously?
But I do agree with your expected utility calculations. The problem is, I can’t motivate myself to write under conditions of uncertainty like that. I understand that shouldn’t matter, especially since I consider such a project worthwhile even in the absence of $155, but it does.
Getting it done to other’s satisfaction and getting it done to your own are not mutually exclusive. You can work quickly to win the prize and then go back and expand.
No, I can’t. That’s my problem, as I’ve been explaining in my comments.
Perhaps you two should collaborate and split the prize in some way.
I think it’s a bit late for that. But I’ve PMed Duke.
Could you specify the drawbacks you see to a race? Does something else seem like it would work better? Does a dated prize (where the best entry is selected on a specific date) or a work contract seem like it would work significantly better? I’m Just Trying Things right now, and I’m not especially attached to this format.
Which are the especially hard questions? What is the primary source of difficulty?
Duplicated effort, some of which is presumably invisible.
From what I remember of the prize literature, prizes work best for well-delineated, objectively judgeable accomplishments. Like putting a rocket into orbit—you can plan out what is necessary, and it’s clear when you succeeded. A literature review isn’t really either. So I’m not sure a prize in general is succeeded for the task.
My personal problems are that:
I can’t bring myself to start a new project like intensively researching spaced repetition to establish exactly how good the evidence is, how effective it is, and what domains it does and does not work in, without having a good idea of what I might receive in turn. Uncertainty is paralyzing for me.
This is especially true because your prize is large—I felt guilty taking a bitcoin from Kiba for Bitcoin is Worse is Better because I knew it was incomplete in several respects, although it’s one of my best essays and gave him a lot of traffic; and a bitcoin is only worth $20, so how would I feel delivering something bad for $155? I’d feel really terrible!
The difficult questions are:
How much does spaced repetition actually help memory?
Quantifying the advantage in any more than an ad hoc way like http://www.gwern.net/Mnemosyne#fn3 would be difficult since it requires delving into the specifics of the data, and forgetting curves are non-linear so it’s not clear what is even meant by ‘how much’ - memories per review at 1 year? At 10 years? At 50 years? Integrated over an actuarially projected lifespan?
Does spaced repetition have hidden benefits or costs?
There’s probably no direct research on this, so now one has to delve into related research fields and look for very weak evidence or correlations, like maybe high memory correlated with longevity in old people.
Is there any interesting ongoing research? If so, what is it on?
New research tends to be hard to get (= wasted time going around paywalls) and requires following a lot of citations in bibliographies and checking out blogs and emails to really get a sense of where a field is now.
The best way is probably to informally or formally contract with someone like lukeprog (who does this sort of literature review routinely) to conduct the review.
Great comment. Perhaps I should be reading the prize literature! I’ll think about how to structure this better.
On your personal problems: consider finding someone who has more time and acting as a consultant for them. For example, Duke, who comments elsewhere in this thread.
Just to make sure I’m clear, I’m not trying to gain a detailed or very specific understanding of the literature on SR, but to figure out how confident I should be in recommending SR to others and to myself and what I should recommend others and I do and what I should expect it to help and how much.
I actually hadn’t read your page before, so thanks. I actually think your page is a fairly good start.
Do you have a good starting point for reading on the prize literature?
Hm, not really. What I would do is google Robin Hanson, Anders Broberg, and LW for mentions of ‘prize’ and use them as entrance into the academic literature, and spider authors & citations & bibliographies from there.
ok, thanks.
It’s not a literature review, but http://33bits.org/2011/06/06/the-surprising-effectiveness-of-prizes-as-catalysts-of-innovation/ does link to a McKinsey report on prizes, which ought to be good reading if their reputation can be relied upon.
It’s funny you linked to my post on prizes, which brought me here, because I’m actually very interested in spaced repetition. In fact just a couple of weeks ago I mentioned it’s potentially worth millions of dollars to me: http://arvindn.livejournal.com/132233.html
You guys need to get more rational about the value and importance of what you’re doing in this community. You’re arguing over trivial amounts of money here. I’d say that any post that’s good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.
I’m putting my money where my mouth is chipping in $100 for the new prize. [Edit. Hmm, I got a paypal receipt but the total on chipin hasn’t changed. Maybe just need to wait for a bit.]
Finally, I’m only a sporadic reader, but allow me to express my deep gratitude and admiration for the community.
! You’re Arvind? I had no idea. Gosh, the past few days have been interesting for me—first Piotr Wozniak tells me he’s incrementally reading 8 articles mentioning or by me, then Danny Hillis replies to a comment of mine over at the Technium, and now here’s Arvind Narayanan.
On one hand, I agree—if a main page post improves one high-value person’s time, that alone is probably a few tens of thousands’ worth. The other part of me is cynically remarking that Wozniak and psychologists in general have been advocating spaced repetition for the last >2 decades / centuries (respectively), the research is unassailable, and yet… So what’s the marginal return on another literature survey?
A few comments:
You mention creatine on your blog. Creatine only works in specific demographics; I don’t know whether they include you.
You mention dual n-back; the results seem to keep getting weaker since Jaeggi 2008, and Jaeggi 2011 seems extremely weak. I used to be extremely enthusiastic, but as the studies keep coming in… At this point, I’m not sure how much time is worth putting into DNB for just the WM benefits. Probably not much past D5B though.
Thanks for the information! That does look pretty damning for dual n-back. As for creatine, I’m not vegetarian but my meat consumption is lower than average, especially when I’m not weight training. At any rate, I stopped taking it after a couple of months because I decided the unknown long-term health risk from increased renal stress wasn’t worth it.
Note that posts aren’t voted up to anywhere; they’re promoted by editors.
Ah, I see, thanks.
For relevant research, see all the sources I listed here.
I’m not quite willing to chipin for the individual in question. But I’m happy to donate $20 to the charity of their choice.
Can you explain your reasons?
It is a compromise between fuzzies v. utilons because I am unable to purchase them separately along with some other constraints. If the money goes to a charity I get more fuzzies, and given that whoever would do this would likely pick a charity that I consider to be a worthwhile charity (given the general framework of LW) there’s a high chance that it will produce heavy utilons. I also don’t have that much money so I can convince myself to part with more if I know that it is going to a charity rather than going to someone’s personal use.
Thanks :)
My entry has been submitted: http://lesswrong.com/lw/64k/memory_spaced_repetition_and_life/
I chipped in $10 for the winner of the prize.
My sincere thanks to anyone who participates!
I am an Anki user and I am interested in working on this project.
I am working on this vigorously and expect to produce a submitted product soon. Proceed at your own risk if you are considering competing for this prize.
Can you describe what does and does not increase your motivation? Does more money increase/decrease? The race aspect? The competitive aspect?
the answers to your first few questions seem to unfortunately be “it depends”. the point of SRS is that you review when you are on the cusp of forgetting a piece of knowledge, which varies drastically depending on individual, conditions, and topic.
Cross-commented from submission thread:
I think it is unfair of you to post a public critique of my submission since this is a contest. I have effectively been penalized for being first. Every submission that follows will have the benefit of seeing this critique.
I am also concerned that you have decided to change the contest format immediately following my submission. In my estimation, you had either already decided to change the format prior to my submission (clearly a major disadvantage to me), or you decided to change the format based on my submission, which, again, effectively penalizes me for being first.
Additionally, I request arbitration. I think this contest has been mismanaged to my detriment; as such, I am entitled to financial compensation.
Link
After some deliberation, I’ve decided to withdraw this request. I am content with my submission. I am also content not to receive the prize or any portion thereof.