A Year of Spaced Repetition Software in the Classroom

Last year, I asked LW for some ad­vice about spaced rep­e­ti­tion soft­ware (SRS) that might be use­ful to me as a high school teacher. With said ad­vice came a re­quest to write a fol­low-up af­ter I had ac­cu­mu­lated some ex­pe­rience us­ing SRS in the class­room. This is my re­port.

Please note that this was not a sci­en­tific ex­per­i­ment to de­ter­mine whether SRS “works.” Prior stud­ies are already pretty con­vinc­ing on this point and I couldn’t think of a prac­ti­cal way to run a con­trol group or “blind” my­self. What fol­lows is more of an in­for­mal de­briefing for how I used SRS dur­ing the 2014-15 school year, my in­sights for oth­ers who might want to try it, and how the ex­pe­rience is chang­ing how I teach.

Summary

SRS can raise stu­dent achieve­ment even with stu­dents who won’t use the soft­ware on their own, and even with fre­quent dis­rup­tions to the study sched­ule. Gains are most ap­par­ent with the already high-perform­ing stu­dents, but are also mean­ingful for the low­est stu­dents. De­liber­ate efforts are needed to get stu­dent buy-in, and get­ting the most out of SRS may re­quire changes in course de­sign.

The software

After look­ing into var­i­ous pro­grams, in­clud­ing the game-like Mem­rise, and even writ­ing my own sim­ple SRS, I ul­ti­mately went with Anki for its multi-plat­form availa­bil­ity, cloud sync, and ease-of-use. I also wanted a pro­gram that could act as an im­promptu catch-all bin for the 2,000+ cards I would be pro­duc­ing on the fly through­out the year. (Mem­rise, in con­trast, re­ally needs clearly defined units pack­aged in ad­vance).

The students

I teach 9th and 10th grade English at an above-av­er­age sub­ur­ban Amer­i­can pub­lic high school in a be­low-av­er­age state. Mine are the lower “re­quired level” stu­dents at a school with high en­rol­l­ment in hon­ors and Ad­vanced Place­ment classes. Gen­er­ally speak­ing, this means my stu­dents are mostly not self-mo­ti­vated, are only very weakly mo­ti­vated by grades, and will not do any­thing school-re­lated out­side of class no mat­ter how much it would be in their in­ter­est to do so. There are, of course, plenty of ex­cep­tions, and my stu­dents span an ex­tremely wide range of abil­ity and ap­a­thy lev­els.

The procedure

First, what I did not do. I did not make Anki decks, as­sign them to my stu­dents to study in­de­pen­dently, and then quiz them on the con­tent. With hon­ors classes I taught in pre­vi­ous years I think that might have worked, but I know my cur­rent stu­dents too well. Only about 10% of them would have done it, and the rest would have blamed me for their failing grades—with some jus­tifi­ca­tion, in my opinion.

In­stead, we did Anki to­gether, as a class, nearly ev­ery day.

As ini­tial setup, I cre­ated a sep­a­rate Anki pro­file for each class pe­riod. With a third-party add-on for Anki called Zoom, I en­larged the dis­play font sizes to be clearly leg­ible on the in­ter­ac­tive white­board at the front of my room.

Nightly, I wrote up cards to re­in­force new ma­te­rial and in­te­grated them into the deck in time for the next day’s classes. This av­er­aged about 7 new cards per les­son pe­riod.Th­ese cards came in many va­ri­eties, but the three main types were:

  1. con­cepts and terms, of­ten with re­versed com­pan­ion cards, some­times sup­ple­mented with “what is this an ex­am­ple of” sce­nario cards.

  2. vo­cab­u­lary, 3 cards per word: word/​def, re­verse, and fill-in-the-blank ex­am­ple sentence

  3. gram­mar, usu­ally in the form of “What change(s), if any, does this sen­tence need?” Alter­na­tive cards had differ­ent per­mu­ta­tions of the sen­tence.

Weekly, I up­dated the deck to the cloud for self-mo­ti­vated stu­dents wish­ing to study on their own.

Daily, I led each class in an Anki re­view of new and due cards for an av­er­age of 8 min­utes per study day, usu­ally as our first ac­tivity, at a rate of about 3.5 cards per minute. As each card ap­peared on the in­ter­ac­tive white­board, I would read it out loud while stu­dents will­ing to share the an­swer raised their hands. Depend­ing on the card, I might offer ad­di­tional time to think be­fore call­ing on some­one to an­swer. Depend­ing on their an­swer, and my im­pres­sions of the class as a whole, I might elab­o­rate or offer some re­minders, mnemon­ics, etc. I would then quickly poll the class on how they felt about the card by hav­ing them show a color by way of a small piece of card-stock di­vided into green, red, yel­low, and white quad­rants. Based on my own judg­ment (in­formed only partly by the poll), I would choose and press a re­sponse but­ton in Anki, de­ter­min­ing when we should see that card again.

End-of-year summary for one of my classes

[Data shown is from one of my five classes. We didn’t start us­ing Anki un­til a cou­ple weeks into the school year.]

Op­por­tu­nity costs

8 min­utes is a sig­nifi­cant por­tion of a 55 minute class pe­riod, es­pe­cially for a teacher like me who fills ev­ery one of those min­utes. Some­thing had to give. For me, I en­tirely cut some va­ri­eties of writ­ten vo­cab re­in­force­ment, and re­duced the time we spent play­ing the team-based vo­cab/​term re­view game I wrote for our in­ter­ac­tive white­boards some years ago. To a lesser ex­tent, I also cut back on some oral read­ing com­pre­hen­sion spot-checks that ac­com­pany my whole-class read­ing ses­sions. On bal­ance, I think Anki was a much bet­ter way to spend the time, but it’s com­pli­cated. Keep read­ing.

Whole-class SRS not ideal

Every stu­dent is differ­ent, and would get the most out of hav­ing a per­sonal Anki pro­file de­ter­mine when they should see each card. Also, most in­di­vi­d­u­als could study many more cards per minute on their own than we av­er­aged do­ing it to­gether. (To be fair, a small hand­ful of my stu­dents did use the soft­ware in­de­pen­dently, judg­ing from Ankiweb down­load stats)

Get­ting stu­dent buy-in

Be­fore we started us­ing SRS I tried to sell my stu­dents on it with a heart­felt, over-pre­pared 20 minute pre­sen­ta­tion on how it works and the su­per­pow­ers to be gained from it. It might have been a waste of time. It might have changed some­one’s life. Hard to say.

As for the daily class re­view, I in­duced en­gage­ment partly through par­ti­ci­pa­tion points that were part of the fi­nal semester grade, and which stu­dents knew I tracked closely. Rais­ing a hand could earn a kind of bonus cur­rency, but was never re­quired—un­like look­ing up front and show­ing col­ors dur­ing polls, which I in­sisted on. When I thought stu­dents were just re­flex­ively hold­ing up the same color and zon­ing out, I would some­times spot check them on the last card we did and pe­nal­ize them if war­ranted.

But be­cause I know my stu­dents are not strongly mo­ti­vated by grades, I think the most im­por­tant in­fluence was my at­ti­tude. I made it a point to re­ally turn up the charm dur­ing re­view and play the part of the en­gag­ing game show host. Pos­i­tive feed­back. Coax­ing out the lurk­ers. Keep­ing that en­ergy up. Be­ing ready to kill and joke about bad cards. Re­mind­ing classes how awe­some they did on tests and as­sign­ments be­cause they knew their Anki stuff.

(This is a good time to point out that the av­er­age re­view time per class pe­riod sta­bi­lized at about 8 min­utes be­cause I tried to end re­views be­fore stu­dent en­gage­ment ta­pered off too much, which typ­i­cally started hap­pen­ing at around the 6-7 minute mark. Oc­ca­sional short end-of-class re­views mostly ac­count for the differ­ence.)

I also got my stu­dents more on the Anki band­wagon by show­ing them how this was di­rectly linked re­duced note-tak­ing re­quire­ments. If I could trust that they would re­mem­ber some­thing through Anki alone, why waste time wait­ing for them to write it down? They were un­likely to study from those notes any­way. And if they aren’t look­ing down at their pa­per, they’ll be pay­ing more at­ten­tion to me. I bet­ter come up with more cool things to tell them!

Mak­ing memories

Every­thing I had read about spaced rep­e­ti­tion sug­gested it was a great re­in­force­ment tool but not a good way to in­tro­duce new ma­te­rial. With that in mind, I tried hard to find or cre­ate mem­o­rable images, ex­am­ples, mnemon­ics, and anec­dotes that my Anki cards could be­come hooks for, and to get those cards into cir­cu­la­tion as soon as pos­si­ble. I even gave this method a mantra: vivid mem­ory, card ready”.

When a stu­dent dur­ing re­view raised their hand, gave me a pained look, and said, “like that time when....” or “I can see that pic­ture of...” as they strug­gled to re­mem­ber, I knew I had done well. (And I would always wait a mo­ment, be­cause they would usu­ally get it.)

Baby cards need im­me­di­ate love

Un­for­tu­nately, if the card wasn’t in­tro­duced quickly enough—within a day or two of the les­son—the en­tire mem­ory of­ten van­ished and had to be recre­ated, kil­ling the mo­men­tum of our re­view. This hap­pened far too of­ten—not be­cause I didn’t write the card soon enough (I stayed re­ally on top of that), but be­cause it didn’t always come up for study soon enough. There were a few rea­sons for this:

  1. We of­ten had too many due cards to get through in one ses­sion, and by de­fault Anki puts new cards be­hind due ones.

  2. By de­fault, Anki only in­tro­duces 20 new cards in one ses­sion (I soon un­capped this).

  3. Some cards were in cat­e­gories that I gave lower pri­or­ity to.

Two ob­vi­ous cures for this prob­lem:

  1. Make fewer cards. (I did get more se­lec­tive as the year went on.)

  2. Have all cards prepped ahead of time and in­tro­duce new ones at the end of the class pe­riod they go with. (For prac­ti­cal rea­sons, not the least of which was the fact that I didn’t always know what cards I was mak­ing un­til af­ter the les­son, I did not do this. I might able to next year.)

Days off suck

SRS is meant to be used ev­ery day. When you take week­ends off, you get a back­log of due cards. Not only do my stu­dents take ev­ery week­end and ma­jor holi­day off (slack­ers), they have a few 1-2 week va­ca­tions built into the cal­en­dar. Com­ing back from a week’s va­ca­tion means a 9-day back­log (due to the week­ends book­end­ing it). There’s no good workaround for stu­dents that won’t study on their own. The best I could do was run longer or mul­ti­ple Anki ses­sions on re­turn days to try catch up with the back­log. It wasn’t enough. The “caught up” con­di­tion was not nor­mal for most classes at most points dur­ing the year, but rather some­thing to as­pire to and oc­ca­sion­ally ap­plaud our­selves for reach­ing. Some cards spent weeks or months on the bot­tom of the stack. Me­mories died. Baby cards emerged stil­lborn. Learn­ing was lost.

Need­less to say, the last weeks of the school year also had a cer­tain silli­ness to them. When the class will never see the card again, it doesn’t mat­ter whether I push the but­ton that says 11 days or the one that says 8 months. (So I re­duced pol­ling and ac­cel­er­ated our cards/​minute rate.)

Never be­fore SRS did I fully ap­pre­ci­ate the loss of learn­ing that must hap­pen ev­ery sum­mer break.

Triage

I kept each course’s mas­ter deck di­vided into a few large sub­decks. This was ini­tially for or­ga­ni­za­tional rea­sons, but I even­tu­ally started us­ing it as a pri­ori­tiz­ing tool. This hap­pened af­ter a curse-wor­thy dis­cov­ery: if you tell Anki to re­view a deck made from sub­decks, due cards from sub­decks higher up in the stack are shown be­fore cards from decks listed be­low, no mat­ter how over­due they might be. From that point, on days when we were back­logged (most days) I would speci­fi­cally re­view the con­cept/​ter­minol­ogy sub­deck for the cur­rent semester be­fore any other sub­decks, as these were my high­est pri­or­ity.

On a cou­ple of oc­ca­sions, I also used Anki’s study deck tools to cre­ate tem­po­rary decks of es­pe­cially high-pri­or­ity cards.

Seiz­ing those moments

Veteran teach­ers start ac­quiring a sense of when it might be a good time to go off book and teach some­thing that isn’t in the unit, and maybe not even in the cur­ricu­lum. Maybe it’s teach­ing ex­actly the right word to de­scribe a vivid situ­a­tion you’re read­ing about, or maybe it’s ad­vice on what to do in a cer­tain type of emer­gency that nearly hap­pened. As the year pro­gressed, I found my­self hu­mor­ing my in­stincts more of­ten be­cause of a new con­fi­dence that I can turn an im­pres­sion­able mo­ment into a strong mem­ory and lock it down with a new Anki card. I don’t even care if it will ever be on a test. This in­sight has me ques­tion­ing a great deal of what I thought knew about or­ga­niz­ing a cur­ricu­lum. And I like it.

A lifeline for low performers

An ac­ci­den­tal dis­cov­ery came from hav­ing writ­ten some cards that were, it was im­me­di­ately ob­vi­ous to me, much too easy. I was em­bar­rassed to even be read­ing them out loud. Then I saw which hands were com­ing up.

In any class you’ll get some small num­ber of ex­tremely low perform­ers who never seem to be do­ing any­thing that we’re do­ing, and, when con­fronted, deny that they have any abil­ity what­so­ever. Some of the hands I was see­ing were at­tached to these stu­dents. And you bet­ter be­lieve I called on them.

It turns out that easy cards are re­ally im­por­tant be­cause they can give wins to stu­dents who des­per­ately need them. Know­ing a 6th grade level card in a 10th grade class is no great achieve­ment, of course, but the ac­tion takes what had been nega­tive morale and nudges it up­ward. And it can trend. I can build on it. A few of these stu­dents started mak­ing Anki the thing they did in class, even if they ig­nored ev­ery­thing else. I can con­fi­dently name one stu­dent I’m sure passed my class only be­cause of Anki. Don’t get me wrong—he just barely passed. Most cards re­mained over his head. Anki was no mir­a­cle cure here, but it gave him and I some­thing to work with that we didn’t have when he failed my class the year be­fore.

A spring­board for high achievers

It’s not even fair. The low­est stu­dents got some­thing im­por­tant out of Anki, but the high­est achiev­ers drank it up and used it for rocket fuel. When peo­ple ask who’s widen­ing the achieve­ment gap, I guess I get to raise my hand now.

I re­fuse to feel bad for this. Smart kids are badly un­der­served in Amer­i­can pub­lic schools thanks to poli­cies that en­courage staff to fo­cus on that slice of stu­dents near (but not at) the bot­tom—the ones who might just barely be able to pass the state test, given enough at­ten­tion.

Where my bright stu­dents might have been used to high Bs and low As on tests, they were now break­ing my scales. You could see it in the mul­ti­ple choice, but it was most ob­vi­ous in their writ­ing: they were skil­lfully work­ing in ter­minol­ogy at an un­prece­dented rate, and mak­ing way more at­tempts to use new vo­cab­u­lary—at­tempts that were, for the most part, suc­cess­ful.

Given the seem­ingly ob­jec­tive na­ture of Anki it might seem coun­ter­in­tu­itive that the benefits would be more ob­vi­ous in writ­ing than in mul­ti­ple choice, but it ac­tu­ally makes sense when I con­sider that even with­out SRS these stu­dents prob­a­bly would have known the terms and the vo­cab well enough to get mul­ti­ple choice ques­tions right, but might have lacked the con­fi­dence to use them on their own ini­ti­a­tive. Anki gave them that ex­tra con­fi­dence.

A wash for the ap­a­thetic mid­dle?

I’m con­fi­dent that about a third of my stu­dents got very lit­tle out of our Anki re­view. They were ei­ther re­ally good at fak­ing in­volve­ment while they zoned out, or didn’t even try to pre­tend and just took the hit to their par­ti­ci­pa­tion grade day af­ter day, no mat­ter what I did or who I con­tacted.

Th­ese weren’t even nec­es­sar­ily failing stu­dents—just the ap­a­thetic mid­dle that’s smart enough to re­mem­ber some frac­tion of what they hear and re­gur­gi­tate some frac­tion of that at the ap­pro­pri­ate times. Re­view of any kind holds no in­ter­est for them. It’s a re­run. They don’t re­ally know the ma­te­rial, but they tell them­selves that they do, and they don’t care if they’re wrong.

On the one hand, these stu­dents are no worse off with Anki than they would have been with with the ac­tivi­ties it re­placed, and no­body cries when av­er­age kids get av­er­age grades. On the other hand, I’m not ok with this… but so far I don’t like any of my ideas for what to do about it.

Put­ting up num­bers: a case study

For un­planned rea­sons, I taught a unit at the start of a quar­ter that I didn’t for­mally test them on un­til the end of said quar­ter. His­tor­i­cally, this would have been a dis­aster. In this case, it worked out well. For five weeks, Anki was the only on­go­ing ex­po­sure they were get­ting to that unit, but it proved to be enough. Be­cause I had given the same test as a pre-test early in the unit, I have some num­bers to back it up. The test was all mul­ti­ple choice, with two sec­tions: the first was on gen­eral ter­minol­ogy and con­cepts re­lated to the unit. The sec­ond was a much harder read­ing com­pre­hen­sion sec­tion.

As ex­pected, scores did not go up much on the read­ing com­pre­hen­sion sec­tion. Over­all read­ing lev­els are very difficult to boost in the short term and I would not ex­pect any one unit or quar­ter to make a sig­nifi­cant differ­ence. The av­er­age score there rose by 4 per­centage points, from 48 to 52%.

Scores in the ter­minol­ogy and con­cept sec­tion were more en­courag­ing. For ma­te­rial we had not cov­ered un­til af­ter the pre-test, the av­er­age score rose by 22 per­centage points, from 53 to 75%. No sur­prise there ei­ther, though; it’s hard to say how much credit we should give to SRS for that.

But there were also a num­ber of ques­tions about ma­te­rial we had already cov­ered be­fore the pretest. Be­ing the ear­liest ma­te­rial, I might have ex­pected some degra­da­tion in perfor­mance on the sec­ond test. In­stead, the already strong av­er­age score in that sec­tion rose by an ad­di­tional 3 per­centage points, from 82 to 85%. (Th­ese num­bers are less re­li­able be­cause of the smaller num­ber of ques­tions, but they tell me Anki at least “locked in” the older knowl­edge, and may have strength­ened it.)

Some other time, I might try re­serv­ing a sec­tion of con­tent that I teach be­fore the pre-test but don’t make any Anki cards for. This would give me a way to com­pare Anki to an al­ter­na­tive re­view ex­er­cise.

What about for­mal stan­dard­ized tests?

I don’t know yet. The scores aren’t back. I’ll prob­a­bly be shown some “value added” anal­y­sis num­bers at some point that tell me whether my stu­dents beat ex­pec­ta­tions, but I don’t know how much that will tell me. My stu­dents were con­sis­tently beat­ing ex­pec­ta­tions be­fore Anki, and the state gave an en­tirely differ­ent test this year be­cause of leg­is­la­tive changes. I’ll go back and re­vise this para­graph if I learn any­thing use­ful.

Those dis­cus­sions...

If I’m try­ing to ac­quire a new skill, one of the first things I try to do is listen to skil­led prac­ti­tion­ers of that skill talk about it to each other. What are the terms-of-art? How do they use them? What does this tell me about how they see their craft? Their short­hand is a trea­sure trove of crys­tal­lized con­cepts; once I can use it the same way they do, I find I’m work­ing at a level of ab­strac­tion much closer to theirs.

Similarly, I was hop­ing Anki could help make my stu­dents more fluent in the sub­ject-spe­cific lex­i­con that helps you score well in an­a­lyt­i­cal es­says. After in­tro­duc­ing a new term and mak­ing the Anki card for it, I made ex­tra efforts to use it con­ver­sa­tion­ally. I used to shy away from that be­cause so many stu­dents would have for­got­ten it im­me­di­ately and tuned me out for not mak­ing any sense. Not this year. Once we’d seen the card, I used the term freely, with only the oc­ca­sional re­minder of what it meant. I started us­ing mul­ti­ple terms in the same sen­tence. I started talk­ing about writ­ing and anal­y­sis the way my fel­low ex­perts do, and so in­vited them into that world.

Even though I was already see­ing writ­ten ev­i­dence that some of my high perform­ers had as­similated the lex­i­con, the high qual­ity dis­cus­sions of these same stu­dents caught me off guard. You see, I usu­ally dread whole-class dis­cus­sions with non-hon­ors classes be­cause good com­ments are so rare that I end up de­ject­edly spout­ing all the in­sights I had hoped they could find. But by the end of the year, my stu­dents had stepped up.

I think what hap­pened here was, as with the writ­ing, as much a boost in con­fi­dence as a boost in fluency. What­ever it was, they got into some good dis­cus­sions where they used the ter­minol­ogy and built on it to say smarter stuff.

Don’t get me wrong. Most of my stu­dents never got to that point. But on av­er­age even small groups with­out smart kids had a no­tice­ably higher level of dis­course than I am used to hear­ing when I break up the class for smaller dis­cus­sions.

Limitations

SRS is in­her­ently weak when it comes to the ab­stract and com­plex. No card I’ve de­vised en­ables a stu­dent to de­velop a dis­tinc­tive au­tho­rial voice, or write es­say open­ings that re­veal just enough to make the reader cu­ri­ous. Yes, you can make cards about strate­gies for this sort of thing, but these were con­sis­tently my worst cards—the overly difficult “leeches” that I even­tu­ally sus­pended from my decks.

A less ob­vi­ous limi­ta­tion of SRS is that stu­dents with a very strong grasp of a con­cept of­ten fail to ap­ply that knowl­edge in more au­then­tic situ­a­tions. For in­stance, they may know perfectly well the differ­ence be­tween “there”, “their”, and “they’re”, but never pause to think care­fully about whether they’re us­ing the right one in a sen­tence. I am very open to sug­ges­tions about how I might train my stu­dents’ au­tonomous “Sys­tem 1″ brains to have “in­ter­rupts” for that sort of thing… or even just a re­flex to go back and check af­ter finish­ing a draft.

Mov­ing forward

I ab­solutely in­tend to con­tinue us­ing SRS in the class­room. Here’s what I in­tend to do differ­ently this com­ing school year:

  • Re­duce the num­ber of cards by about 20%, to maybe 850-950 for the year in a given course, mostly by re­duc­ing the num­ber of vari­a­tions on some over­ex­posed con­cepts.

  • Be more will­ing to add ex­tra Anki study ses­sions to stay bet­ter caught-up with the deck, even if this means my les­son con­tent doesn’t line up with class pe­ri­ods as neatly.

  • Be more will­ing to press the red but­ton on cards we need to re-learn. I think I was too hes­i­tant here be­cause we were rarely caught up as it was.

  • Re­work un­der­perform­ing cards to be sim­pler and more fun.

  • Use more sim­ple cloze dele­tion cards. I only had a few of these, but they worked bet­ter than I ex­pected for struc­tured idea sets like, “char­ac­ter­is­tics of a tragic hero”.

  • Take a less lin­ear and more op­por­tunis­tic ap­proach to in­tro­duc­ing terms and con­cepts.

  • Allow for more im­promptu dis­cus­sions where we bring up older con­cepts in rele­vant situ­a­tions and build on them.

  • Shape more of my les­sons around the vivid mem­ory, card ready” philos­o­phy.

  • Con­tinue to re­duce need­less stu­dent note-tak­ing.

  • Keep a close eye on 10th grade stu­dents who had me for 9th grade last year. I won­der how much they re­tained over the sum­mer, and I can’t wait to see what a sec­ond year of SRS will do for them.

Sugges­tions and com­ments very wel­come!