Very specifically, perhaps one of the most useful results in the psychology of learning and memory is the spacing effect, and by extension, spaced repetition. In that regard:
Now the part wherein I question your premise: a study of learning is certainly useful, but as so many are apt to mention, one can be an effective learner who is studying a sample that is largely noise, and in a contest, you might be defeated by a less effective learner who is studying a better sample. For the same meta-reason that you concluded that it’s useful to understand how to learn, it’s useful to have learning resources that are correct and efficiently presented, and moving one level up, how to find such learning resources in the first place when they have not yet been conveniently collected:
Going even further up, there are things that affect your actual progress, such as stress, sleep, diet, health both physical and psychological, and akrasia. Piotr Wozniak has written up a nice summary in that regard.
And lastly, there is the meta-trap itself, which you need only worry about if your interests lie outside the psychology of learning.
Good links (also, very intelligent question from Bound_up). I’ll present a contrarian perspective I’m mildly confident in. I tried Anki and found it to be lots of work with a small payoff. In an age where you can instantly look up anything on the internet, spending a full five minutes cumulatively over the course of your life to make one particular fact unnecessary to google seems dubious. My Anki reviewing habit was stable for maybe 4-6 months after I installed it (I was habitually reviewing Anki while on public transportation, but got sick and decided this was too stressful.) I find it suspicious that so few people seem to use Anki well and consistently; I do know of a few, but if it was actually a slam dunk I would expect to see more results from people than I do at this point. See more: http://lesswrong.com/lw/juq/a_vote_against_spaced_repetition/
So what can you do instead of spaced repetition? It seems pretty clear that we forget most of what we read; I read Benjamin Franklins’ autobiography as a teenager and felt very proud of myself; years later if I was to write down everything I could remember, I doubt it would fill a single page. Was there any point in reading it? I’m not sure, and my current guess is complicated.
To figure out how to learn more effectively, we could start by asking what sort of payoffs we expect from learning. Here are some payoffs that come to me off the top of my head:
Sometimes I’m learning something because I expect to apply the knowledge right away to some problem or project that I’m working on. The payoff in these cases is sure and immediate. Additionally, through applying my knowledge and learning by doing, I typically walk away with a deeper understanding. To adopt this as your learning strategy, make a deal with yourself that you won’t try to learn anything that seems useless, and in exchange, you’ll always be audacious enough to tackle difficult projects that seem interesting and important to you, even if they seem way beyond your capabilities, because you’ll learn your way there. Your goal is no longer “learn about machine learning”, it’s “win a Kaggle competition” or “find a pattern in stock market data that lets you make money” (both of these have concrete payoffs in the form of resume gardening and cash respectively, so it’s actually pretty sensible to have them as goals; you’d probably want to gradually work up to them with a series of more achievable project-based goals first though). My impression is that hacker type intellects seem especially predisposed to learn well through project-based learning (e.g. one of my brothers built an circuit simulator when he wanted to learn about electronics).
Sometimes I find myself glad that I learned something because it constitutes a fact or model that’s useful for what I’m doing, but I wouldn’t have realized in advance the Google keywords I would need to use in order to solve my problem. (It may seem rather mundane to note what sorts of questions keyword search is and isn’t good for. But consider the limiting case where you could ask Google literally any question (e.g. does P = NP) and it would give you the correct answer. This is an AGI complete problem, so if this were true Google would be a superintelligence and there’d be no more use for humans like you. Your job as a human is to answer questions Google can’t. Memorizing state capitals is totally pointless, but having a grab bag of mathematical, statistical, economic, and computational models at your disposal is a good use of time. Google can’t tell you whether to model your thing as a quadratic or an exponential. It’s not clear to me how deep ones knowledge of these models even needs to be; as Steve Yegge writes in this brilliant post, just knowing the name of the branch of mathematics that has the solution to your problem may be enough, at which point you can bone up on it and then apply it to your problem. My guess is that deep knowledge will let you see some isomorphisms that shallow knowledge doesn’t get you, and deep knowledge is typically required to do cutting edge work in a field instead of just applying its results in another field. Deep knowledge of multiple interacting fields that are rarely combined in a single person probably pushes you in to superpower territory.
Related to the above, new facts can inform your thinking at a subconscious level that’s hard to describe. In addition to a historian being able to point to specific un-google-able historical situations analogous to one that’s unfolding, they may be able to make better intuitive predictions even without citing specific examples (although I guess Tetlock’s work on expert predictions suggests that the historian’s opinion won’t be very predictive either way). In some cases these model changes may actually be harmful; reading the news will cause you to overestimate the rate of newsworthy disasters; reading psychology studies that are selected for interestingness (and thus counterintuitiveness) may lead you to believe humans are more counterintuitive than they actually are. But in the best case this can be very powerful. I had a great math teacher who would challenge students to solve new math problems without yet having been exposed to the standard method used for them, and I feel like this (along with studying counterintuitive math results like the Monty Hall problem) taught my brain this weird skill of turning off my intuition and demanding that everything be modeled rigorously and mechanistically, which feels like a valuable mental muscle to have.
In general, I wonder how your strategies might differ depending on whether you’re more interested in creating new knowledge vs applying existing knowledge in a straightforward way. Creating new knowledge probably requires more creativity, and it may be sensible to have explicit creativity strategies that your learning process plugs in to. For example, Richard Feynman supposedly memorized a list of a dozen or so interesting unsolved problems and whenever he came across some new mathematical trick, he’d test it against each of his problems, occasionally scoring a hit and wowing people with his “genius”. So you could imagine keeping lists of mathematical tricks and unsolved problems as you read, say, in order to facilitate this. There are other resources like Edward de Bono’s books or The Creative Habit on systematizing creativity. The steelman of using Anki might be that it makes all the concepts you’re learning highly cognitively available, thus giving you the opportunity to see more applications for them. Under this view you might create cards specifically for ideas that seemed important but not especially salient/memorable/frequently encountered.
In general I think the problem of knowing what sort of information you wouldn’t have known to Google for is a hard one. With my digital notebook, I’ve tried to build an index of information I read by the situation I would use it in, with each page corresponding to some situation. Over time, you can imagine forming pages that act as guides in particular situations; you could imagine a “how to solve tricky math problems” guide you compose for yourself; it’s probably quicker than memorizing the information with Anki; it’s more likely to actually get used appropriately, and it’s more durable and shareable. Or you could imagine an introspection guide with a list of all the major psychological and neurological models you’re familiar with so you can sweep through them and figure out which provides the most self-understanding (is this one for near/far or systems 1 and 2?). I expect creating these sort of guides will also sometimes cause you to remap your understanding in a way you didn’t expect, creating new knowledge, and at the very least will give you the opportunity to deepen your understanding instead of just nodding along with what you read. (nodding along seems OK for repetitive learning resources of low information density, since the repetition will hammer the info in to your head for you. But for highly info dense resources you should be spending half or more of your time trying to integrate what you’re reading with what you already know; staring in to space and figuring out why this knowledge might be surprising seems to work well for this.)
One final piece of advice: maybe intermix your learning about learning with learning some object-level thing, in order to make theory and practice maximally interconnected. For example, do learning study until you think of a learning experiment for today, then take a break, come back, and learn object level stuff for the rest of the day, trying out your experiment (and being willing to abandon it quickly and go back to your accumulated best practices if it isn’t working).
As another person who’s used Anki for quite some time (~ 2 years), my experience agrees with eeuuah. I would also add exceptions to “just Google it.”
It’s easier to maintain knowledge than to reacquire it. The prototypical example here is tying a tie. Having a card that says “tie a four-in-hand knot”, and having to do that occasionally, turns out to be a lot easier than Googling how to tie a tie, especially if you do it infrequently enough that you need to re-learn it every time.
You need to maintain working memory. The prototypical example here is math. Sure, I can look up the definition of an affine subset, but if I’m in the middle of a proof and I need to prove X is an affine subset of V and then need to look up the definition of affine subset, then I suffer a break in my working memory, which sets me back quite a bit.
You need to remember that the fact exists. The prototypical example here is theorems. Being able to Google the Law of Total Probability doesn’t help if I don’t remember that it exists, and it doesn’t tell me when I can apply it. Having an Anki card for Law of Total Probability does both these things.
You need knowledge in a context where you can’t use Google. The prototypical example here is school. Even outside of school, though, there’s situations where it just won’t do to pull out your phone to Google something.
Overall I agree with your post. As someone who feels like they’re getting a lot out of anki, a few quick notes on my experience with it (been using for ~15 months continuously now)
The first 2-4 months of use for me were very difficult, and consisting mostly of making bad cards that didn’t usefully cement knowledge. I quit (for about 6ish months each time) twice.
Anki is much better suited to some knowledge domains than others. I think the classic example of this is language learning—many people undisputedly have a lot of success with srs in this domain.
Your steelman of using anki to make concepts highly available is something I use it for. I’ve installed a number of triggers where starting to think down a certain line brings up “autocompletions” that point me in a useful direction.
The common srs advice to understand before memorizing is absolutely true. Not to harp on the point, but don’t underestimate the importance of this.
I put key points of a lot of workflows into anki. I wasn’t sure this was going to work out (and it didn’t 100%, but I’ve fine tuned my process based on results), but it’s been very valuable in reducing warm up times when getting back into workflows I used to do very regularly that I haven’t need for 6-8 months.
Finally, anki is absolutely the wrong knowledge store for a lot of stuff. While there are many facts I think I am saving on by spending ~5 minutes over my lifetime (at a slight discount, since anki cuts mostly into nonproductive time while lookups cut mostly into maximally productive time), most aren’t. Computers enable big, searchable knowledge stores which is highly valuable. Evernote is in this category, although it didn’t really appeal to me. Gmail archives are another example. I’ve been using and enjoying workflowy for this lately.
Well this turned into a wall of text, so I hope someone can get some benefit from it :)
Note that anki may work crappily for you, but the general theory that people will remember morr things if they’re repeatedly exposed to them likely still applies—just find a method that better suits you.
Some autodidacticism resources on LessWrong:
A brief summary of effective study methods
Book Review: How Learning Works
On learning difficult things
How my math skills improved dramatically
Improving Enjoyment and Retention Reading Technical Literature
Very specifically, perhaps one of the most useful results in the psychology of learning and memory is the spacing effect, and by extension, spaced repetition. In that regard:
gwern’s writings on spaced repetition
Spaced repetition software
Anki
Mnemosyne
Knowledge structuring and representation in learning based on active recall (Pop science version)
Now the part wherein I question your premise: a study of learning is certainly useful, but as so many are apt to mention, one can be an effective learner who is studying a sample that is largely noise, and in a contest, you might be defeated by a less effective learner who is studying a better sample. For the same meta-reason that you concluded that it’s useful to understand how to learn, it’s useful to have learning resources that are correct and efficiently presented, and moving one level up, how to find such learning resources in the first place when they have not yet been conveniently collected:
Scholarship: How to Do It Efficiently and this comment therein
References & Resources for LessWrong
The Best Textbooks on Every Subject
Khan Academy
Going even further up, there are things that affect your actual progress, such as stress, sleep, diet, health both physical and psychological, and akrasia. Piotr Wozniak has written up a nice summary in that regard.
And lastly, there is the meta-trap itself, which you need only worry about if your interests lie outside the psychology of learning.
Good links (also, very intelligent question from Bound_up). I’ll present a contrarian perspective I’m mildly confident in. I tried Anki and found it to be lots of work with a small payoff. In an age where you can instantly look up anything on the internet, spending a full five minutes cumulatively over the course of your life to make one particular fact unnecessary to google seems dubious. My Anki reviewing habit was stable for maybe 4-6 months after I installed it (I was habitually reviewing Anki while on public transportation, but got sick and decided this was too stressful.) I find it suspicious that so few people seem to use Anki well and consistently; I do know of a few, but if it was actually a slam dunk I would expect to see more results from people than I do at this point. See more: http://lesswrong.com/lw/juq/a_vote_against_spaced_repetition/
So what can you do instead of spaced repetition? It seems pretty clear that we forget most of what we read; I read Benjamin Franklins’ autobiography as a teenager and felt very proud of myself; years later if I was to write down everything I could remember, I doubt it would fill a single page. Was there any point in reading it? I’m not sure, and my current guess is complicated.
To figure out how to learn more effectively, we could start by asking what sort of payoffs we expect from learning. Here are some payoffs that come to me off the top of my head:
Sometimes I’m learning something because I expect to apply the knowledge right away to some problem or project that I’m working on. The payoff in these cases is sure and immediate. Additionally, through applying my knowledge and learning by doing, I typically walk away with a deeper understanding. To adopt this as your learning strategy, make a deal with yourself that you won’t try to learn anything that seems useless, and in exchange, you’ll always be audacious enough to tackle difficult projects that seem interesting and important to you, even if they seem way beyond your capabilities, because you’ll learn your way there. Your goal is no longer “learn about machine learning”, it’s “win a Kaggle competition” or “find a pattern in stock market data that lets you make money” (both of these have concrete payoffs in the form of resume gardening and cash respectively, so it’s actually pretty sensible to have them as goals; you’d probably want to gradually work up to them with a series of more achievable project-based goals first though). My impression is that hacker type intellects seem especially predisposed to learn well through project-based learning (e.g. one of my brothers built an circuit simulator when he wanted to learn about electronics).
Sometimes I find myself glad that I learned something because it constitutes a fact or model that’s useful for what I’m doing, but I wouldn’t have realized in advance the Google keywords I would need to use in order to solve my problem. (It may seem rather mundane to note what sorts of questions keyword search is and isn’t good for. But consider the limiting case where you could ask Google literally any question (e.g. does P = NP) and it would give you the correct answer. This is an AGI complete problem, so if this were true Google would be a superintelligence and there’d be no more use for humans like you. Your job as a human is to answer questions Google can’t. Memorizing state capitals is totally pointless, but having a grab bag of mathematical, statistical, economic, and computational models at your disposal is a good use of time. Google can’t tell you whether to model your thing as a quadratic or an exponential. It’s not clear to me how deep ones knowledge of these models even needs to be; as Steve Yegge writes in this brilliant post, just knowing the name of the branch of mathematics that has the solution to your problem may be enough, at which point you can bone up on it and then apply it to your problem. My guess is that deep knowledge will let you see some isomorphisms that shallow knowledge doesn’t get you, and deep knowledge is typically required to do cutting edge work in a field instead of just applying its results in another field. Deep knowledge of multiple interacting fields that are rarely combined in a single person probably pushes you in to superpower territory.
Related to the above, new facts can inform your thinking at a subconscious level that’s hard to describe. In addition to a historian being able to point to specific un-google-able historical situations analogous to one that’s unfolding, they may be able to make better intuitive predictions even without citing specific examples (although I guess Tetlock’s work on expert predictions suggests that the historian’s opinion won’t be very predictive either way). In some cases these model changes may actually be harmful; reading the news will cause you to overestimate the rate of newsworthy disasters; reading psychology studies that are selected for interestingness (and thus counterintuitiveness) may lead you to believe humans are more counterintuitive than they actually are. But in the best case this can be very powerful. I had a great math teacher who would challenge students to solve new math problems without yet having been exposed to the standard method used for them, and I feel like this (along with studying counterintuitive math results like the Monty Hall problem) taught my brain this weird skill of turning off my intuition and demanding that everything be modeled rigorously and mechanistically, which feels like a valuable mental muscle to have.
In general, I wonder how your strategies might differ depending on whether you’re more interested in creating new knowledge vs applying existing knowledge in a straightforward way. Creating new knowledge probably requires more creativity, and it may be sensible to have explicit creativity strategies that your learning process plugs in to. For example, Richard Feynman supposedly memorized a list of a dozen or so interesting unsolved problems and whenever he came across some new mathematical trick, he’d test it against each of his problems, occasionally scoring a hit and wowing people with his “genius”. So you could imagine keeping lists of mathematical tricks and unsolved problems as you read, say, in order to facilitate this. There are other resources like Edward de Bono’s books or The Creative Habit on systematizing creativity. The steelman of using Anki might be that it makes all the concepts you’re learning highly cognitively available, thus giving you the opportunity to see more applications for them. Under this view you might create cards specifically for ideas that seemed important but not especially salient/memorable/frequently encountered.
In general I think the problem of knowing what sort of information you wouldn’t have known to Google for is a hard one. With my digital notebook, I’ve tried to build an index of information I read by the situation I would use it in, with each page corresponding to some situation. Over time, you can imagine forming pages that act as guides in particular situations; you could imagine a “how to solve tricky math problems” guide you compose for yourself; it’s probably quicker than memorizing the information with Anki; it’s more likely to actually get used appropriately, and it’s more durable and shareable. Or you could imagine an introspection guide with a list of all the major psychological and neurological models you’re familiar with so you can sweep through them and figure out which provides the most self-understanding (is this one for near/far or systems 1 and 2?). I expect creating these sort of guides will also sometimes cause you to remap your understanding in a way you didn’t expect, creating new knowledge, and at the very least will give you the opportunity to deepen your understanding instead of just nodding along with what you read. (nodding along seems OK for repetitive learning resources of low information density, since the repetition will hammer the info in to your head for you. But for highly info dense resources you should be spending half or more of your time trying to integrate what you’re reading with what you already know; staring in to space and figuring out why this knowledge might be surprising seems to work well for this.)
One final piece of advice: maybe intermix your learning about learning with learning some object-level thing, in order to make theory and practice maximally interconnected. For example, do learning study until you think of a learning experiment for today, then take a break, come back, and learn object level stuff for the rest of the day, trying out your experiment (and being willing to abandon it quickly and go back to your accumulated best practices if it isn’t working).
As another person who’s used Anki for quite some time (~ 2 years), my experience agrees with eeuuah. I would also add exceptions to “just Google it.”
It’s easier to maintain knowledge than to reacquire it. The prototypical example here is tying a tie. Having a card that says “tie a four-in-hand knot”, and having to do that occasionally, turns out to be a lot easier than Googling how to tie a tie, especially if you do it infrequently enough that you need to re-learn it every time.
You need to maintain working memory. The prototypical example here is math. Sure, I can look up the definition of an affine subset, but if I’m in the middle of a proof and I need to prove X is an affine subset of V and then need to look up the definition of affine subset, then I suffer a break in my working memory, which sets me back quite a bit.
You need to remember that the fact exists. The prototypical example here is theorems. Being able to Google the Law of Total Probability doesn’t help if I don’t remember that it exists, and it doesn’t tell me when I can apply it. Having an Anki card for Law of Total Probability does both these things.
You need knowledge in a context where you can’t use Google. The prototypical example here is school. Even outside of school, though, there’s situations where it just won’t do to pull out your phone to Google something.
Overall I agree with your post. As someone who feels like they’re getting a lot out of anki, a few quick notes on my experience with it (been using for ~15 months continuously now)
The first 2-4 months of use for me were very difficult, and consisting mostly of making bad cards that didn’t usefully cement knowledge. I quit (for about 6ish months each time) twice.
Anki is much better suited to some knowledge domains than others. I think the classic example of this is language learning—many people undisputedly have a lot of success with srs in this domain.
Your steelman of using anki to make concepts highly available is something I use it for. I’ve installed a number of triggers where starting to think down a certain line brings up “autocompletions” that point me in a useful direction.
The common srs advice to understand before memorizing is absolutely true. Not to harp on the point, but don’t underestimate the importance of this.
I put key points of a lot of workflows into anki. I wasn’t sure this was going to work out (and it didn’t 100%, but I’ve fine tuned my process based on results), but it’s been very valuable in reducing warm up times when getting back into workflows I used to do very regularly that I haven’t need for 6-8 months.
Finally, anki is absolutely the wrong knowledge store for a lot of stuff. While there are many facts I think I am saving on by spending ~5 minutes over my lifetime (at a slight discount, since anki cuts mostly into nonproductive time while lookups cut mostly into maximally productive time), most aren’t. Computers enable big, searchable knowledge stores which is highly valuable. Evernote is in this category, although it didn’t really appeal to me. Gmail archives are another example. I’ve been using and enjoying workflowy for this lately.
Well this turned into a wall of text, so I hope someone can get some benefit from it :)
Note that anki may work crappily for you, but the general theory that people will remember morr things if they’re repeatedly exposed to them likely still applies—just find a method that better suits you.
There is also this Coursera course on “learning how to learn” which I enjoyed a lot: https://www.coursera.org/learn/learning-how-to-learn