It helps to explicitly visualize people who I perceive as being skilled in X failing at it over and over again
Some of the greatest value I’ve gotten out of attending math lectures comes from seeing math Ph.Ds (particularly good ones) make mistakes or even forget exactly how a proof works and have to dismiss class early. It never happened often, but just often enough to keep me from getting discouraged.
Paul Graham writes that studying fields with hard, solved problems (eg mathematics) is useful, because it gives you practice solving hard problems and the approaches and habits of mind that you develop solving those problems are useful when you set out to tackle new (technical) problems. This claim seems at least plausible to me and seems to line up with me personal experience, but you seem like a person who might know why I shouldn’t believe this, so I ask, is there any reason I should doubt that the problem-solving approaches and habits of mind I develop studying mathematics won’t help me as I run into novel technical problems?
If you’re after feedback-for-understanding, providing a student with a list of questions they got wrong and a good solutions manual (which you only have to write once) works most of the time (my guess is around 90% of the time, but I have low confidence in my estimates because I’m capable of successfully working through entire textbooks’ worth of material and needing no human feedback, which I’m told is not often the case). Doing this should be more effective than having the error explained outright a la generation effect.
Another interesting result is that the best feedback for fostering understanding often comes not from experts, who have such a deep degree of understanding and automaticity that it impairs their ability to simulate and communicate with minds struggling with new material, but from students who just learned the material. There’s a risk of students who believe the right thing for the wrong reason propagating their misunderstanding, but I think that pairing up a student who’s struggling with some concept (i.e., throwing a solutions manual at them hasn’t helped them bridge the conceptual gap that caused them to get the question wrong) with a student who understands it is often helpful. IIRC, Sal Khan described using this technique with some success in his book; a friend/mentor who teaches secondary math and keeps up with the literature tells me this works; and I’ve used this basic technique doing an enrichment afterschool program for the local Mathcounts team after the season had ended and can only describe its efficacy as “definitely witchcraft”.
I think there’s a place for graders to give detailed feedback to bad answers, but most of the time, it’s better to force students to do the work themselves and locate their own errors/conceptual gaps, and in most of the remaining cases, to pawn off the responsibility to students (this could be construed as teachers being lazy, but it’s also what, to my knowledge, produces the best learning outcomes). Since detailed feedback is only desirable after two rounds of other approaches that (in my deeply nonrepresentative experience) usually work, I don’t think it makes sense to produce detailed feedback to every wrong answer.
Then again, I don’t fully understand what context you’re thinking in. In my original post, I was thinking about purely diagnostic math tests given to postsecondary students for employers that wouldn’t so much as tell students which questions they got wrong, along the lines of the Royal Statistical Society’s Graduate Diploma (five three-hour tests which grant a credential equivalent to a “good UK honours degree”). In writing this, I’m mostly imagining standardized math tests for secondary students in America (which, I’m given to understand, already have written components), which currently don’t give per-question feedback, but changing that is much less of a pipe dream than creating tests that effectively test understanding. Come to think of it, I think the above approach applies even better to classroom instructors giving their own tests, at either the secondary or postsecondary level.
Tangentially related: the best professor I ever had would type 3–4 pages of general commentary (common errors and why they were wrong and how to do them better, as well as things the class did well) for the class after every problem set and test, generally by the next class. I found this commentary was extraordinarily helpful, not just because of feedback, but because (a) it helped dispel the misperception that everyone else understood everything and I was struggling because I was stupid, (b) taught us to discriminate between bad, mediocre, and good work, and (c) comments like “most of you did [x], which was suboptimal because of [y], but one of you did [z], which takes a bit more work but is a better approach because [~y]” really drove me to not do the minimum amount of work to get an answer when I could do a bit more work to get a stronger solution. (The course was in numerical methods so, as an example, we once had a problem where we had to use some technique where error exploded (I’ve now forgotten since I didn’t have Anki back then) to locate a typo in some numeric data. A sufficient answer would have been to identify the incorrect entry; a stronger answer was to identify the incorrect entry, figure out the error (two digits typed in the wrong order), and demonstrate that fixing the error caused explosions to not happen.)
If we assume that the questions are designed such that a student can answer them upon initial exposure if and only if they deeply understand the material, then the question of identifying graders turns into the much easier question of identifying people who can discriminate between valid and invalid answers. I’m told that being able to discriminate between valid and invalid responses is a necessary condition for subject expertise, so anyone who’s a relevant expert works. One way to demonstrate expertise is by building something that requires expertise. In an extreme example, I’m confident that Grigori Perelman understands topology because he proved the Poincare conjecture, and, for similar reasons, I’m (mostly) confident that Ph.Ds are experts. If we have well-designed tests, we can set the set of people qualified to grade tests as “has built something requiring expertise or has passed a well-designed test graded by someone already in this set.”
It seems conventional wisdom that tests are generally gameable in the sense that an (most?) effective way to produce the best scores involves teaching password guessing rather than actually learning material deeply, i.e. such that the student can use it in novel and useful ways. Indeed, I think this is the case for many (most, even) tests, but also think it possible to write tests that are most easily passed by learning the material deeply. In particular, I don’t see how to game questions like “state, prove, and provide an intuitive justification for Pascal’s combinatorial identity” or “Under what conditions does f(x) = ax^3 + bx^2 + cx + d have only one critical point?″, but that’s more a statement about my mind than the gameability of tests. I would greatly appreciate learning how a test consisting of such questions could be gamed, thereby unlearning an untrue thing; and if no one here can (or, at least, is willing to take the time to) explain how such a thing could be done, well, that’s useful to know, too.
Trying to find the Oxford livestream, I happened across the Saturday Afternoon video.
...And, now it’s private.
For those interested in further reading: Robin Hanson’s take, popularly-written book.
In my high school career, I took precisely one non-honors/AP course when alternatives were present. Recalling my classmates… yeah, I’m now as skeptical as you are.
(I had successfully repressed those memories until now. Thanks so much for the reminder ;)
Any chance your success might influence your colleagues?
Before we started using SRS I tried to sell my students on it with a heartfelt, over-prepared 20 minute presentation on how it works and the superpowers to be gained from it. It might have been a waste of time. It might have changed someone’s life. Hard to say.
I’m less skeptical. You say that you got a few students to use Anki which, while probably not life-changing, is probably significantly life-impacting. If my tenth grade English teacher had introduced Anki to me… well, right now, I’m reteaching myself introductory biology (5 on the AP exam), introductory chemistry (5 on the AP exam) and introductory psychology (A in the college course) because I forgot the content of each of these courses because I lacked Anki. I obviously don’t know everything you do in your classroom, but it’s entirely plausible that, rather than being a waste of time, introducing your students to Anki might have been (on average) the most impactful 20 minutes of teaching you did all year; you just may not see all the benefit in your classroom.
I’m not entirely sure who the audience of this letter is (I’m given to understand “effective altruists” is a pretty heterogeneous group). This affects how your letter should look so much that I can’t give much object-level feedback. For instance, it matters how much of your audience has pre-existing familiarity with things like raising the sanity waterline and rationality as a common interest across causes; if most of them lack this familiarity, I expect they’ll read your first sentence, be unable to bridge an inferential gap, and stop reading.
Ideally, I’d like to know how exactly this letter is getting to its recipients: are you posting on EA forum or mailing it to anyone who’s donated to GiveWell?
Introductory discrete math textbook (pdf) courtesy of MIT. I prefer it to Rosen, which is currently recommended in the MIRI research guide, although I think there exist students who would do better with Rosen’s book.
(How to tell which book you should choose? Well, since this one is Creative Commons, and therefore free, I’d try this one. If you find it’s not saying enough words per theorem, try Rosen. If you think it’s saying too many words per theorem, try these lecture notes. A recommendation to LW’s list of best textbooks is forthcoming, which will contain a complete discussion.)
An earlier version of the book corresponds to these videos lectures, which I find to be excellent, as far as lectures go.
As another person who’s used Anki for quite some time (~ 2 years), my experience agrees with eeuuah. I would also add exceptions to “just Google it.”
It’s easier to maintain knowledge than to reacquire it. The prototypical example here is tying a tie. Having a card that says “tie a four-in-hand knot”, and having to do that occasionally, turns out to be a lot easier than Googling how to tie a tie, especially if you do it infrequently enough that you need to re-learn it every time.
You need to maintain working memory. The prototypical example here is math. Sure, I can look up the definition of an affine subset, but if I’m in the middle of a proof and I need to prove X is an affine subset of V and then need to look up the definition of affine subset, then I suffer a break in my working memory, which sets me back quite a bit.
You need to remember that the fact exists. The prototypical example here is theorems. Being able to Google the Law of Total Probability doesn’t help if I don’t remember that it exists, and it doesn’t tell me when I can apply it. Having an Anki card for Law of Total Probability does both these things.
You need knowledge in a context where you can’t use Google. The prototypical example here is school. Even outside of school, though, there’s situations where it just won’t do to pull out your phone to Google something.
White noise is fine; irrelevant sound effect operates on anything that sounds like it may be human speech, which turns out to be any sort of fluctuating tone.
It has been requested that I post my own take on efficient learning. As I spend half a page describing, this is not yet ready for publishing, but I’m putting out there because there may be (great) benefit to be had. After all, there is low-hanging fruit if you’re willing to abandon traditional methods: simply doing practice problems in a different order may improve your test score by 40 points.
“Baby Rudin” refers to “Principles of Mathematical Analysis”, not “Real and Complex Analysis” (as was currently listed up top.) (Source)
Since this review, Axler has released a third edition. The new edition contains substantial changes (i.e. it’s not the same book being released under “n+1 edition”): though there’s little new material, exercises appear at the end of every section, instead at the end of every chapter, and there’s many more examples given in the body of the text (a longer list of changes can be found on Dr. Axler’s website). I feel these revisions are significant improvements from a pedagogical perspective, as it gives the reader more opportunity to practice prerequisite skills before learning the next thing. The changes also lower the requisite mathematical maturity, which is a good thing (insofar as it makes the book more accessible), although it won’t push the reader to develop mathematical maturity as much. Overall: the third edition came out when I was halfway through the second edition and I felt that the improments merited switching books.
Turns out you’re not the only one who wants to know this. Seems your best bet is to use C-S-v to paste raw text and then format it in the article editor.
Yeah. I’ve taught myself several courses just from textbooks, with much more success than in traditional setups that come with individual attention. I am probably unusual in this regard and should probably typical-mind-fallacy less.
However, I will nitpick a bit. While most textbooks won’t quite have every answer to every question a student could formulate whilst reading it (although the good ones come very close), answers to these questions are typically 30 seconds away, either on Wikipedia or Google. Point about the importance of having people to talk to still stands.
Also, some textbooks (e.g. the AoPS books) have hints for when a student gets stuck on a problem. Point about the importance of having people to help students when they get stuck still stands, although I believe the people best-suited to do this are their classmates; by happy coincidence, these people don’t cost educational organizations anything.
I’m tinkering with a system in which a professor, instead of lecturing, has it as their job to give each of 20 graduate students an hour a week of one-on-one attention (you know, the useful type of individual attention), which the graduate student is expected to prepare for extensively. Similarly, each graduate student is tasked with giving undergraduates 1 hour/week of individual attention. This maintains a professor:student ratio of 200:1 (so MIT needs a grand total of… 57 professors), doesn’t overly burden the mentors, and gives the students much more quality individual attention than I sense they’re currently getting. (Also, I believe that 1 hour of a grad student’s time is going to be more helpful to a student than 1 hour of a professor’s time. Graduate students haven’t become so well-trained in their field they’re no longer able to simulate a non-understanding undergrad in their head (an inability Dr. Mazur claims is shared among lecturers) and I expect there’s benefit from shrinking the age/culture gap. Also, no need to worry about appearing to be the class idiot in front of the person assigning your grade and potentially not giving you the benefit of the doubt on account of being the class idiot.) (Also, it has not escaped my attention that this falls apart at schools that are small or don’t have graduate students. And there’s other problems. Just an idea I’ve had floating around that may be enough in the right direction to effect a positive change.)
Lemma: sum of the degrees of the nodes is twice the number of edges.
Proof: We proceed by induction on the number of edges. If a graph has 0 edges, the the sum of degrees of edges is 0=2(0). Now, by way of induction, assume, for all graphs with n edges, the sum of the degrees of the nodes 2n; we wish to show that, for all graphs with n+1 edges, the sum of the degrees of the nodes is 2(n+1). But the sum of the degrees of the nodes is (2n)+2 = 2(n+1). ∎
The theorem follows as a corollary.
If you want practice proving things and haven’t had much experience so far, I’d recommend Mathematics for Computer Science, a textbook from MIT and distributed under a free license, along with the associated video lectures *. To use Terry Tao’s words, Sipser is writing at both level 1 and 3: he’s giving arguments an experienced mathematician is capable of filling in the details to form a rigorous argument, but also doing so in such a way that a level 1 mathematician can follow along. Critically, however, from what I understand from reading Sipser’s preface, he’s definitely not writing a book to move level 1 mathematicians to level 2, which is a primary goal of the MIT book. If you’re looking to prove things because you haven’t done it much before, I infer you’re essentially looking to transition from level 1 to 2, hence the recommendation.
A particular technique I picked up from the MIT book, which I used here, was that, for inductive proofs, it’s often easier to prove a stronger theorem, since it gives you stronger assumptions in the inductive step.
PM me if you want someone to look over your solutions (either for Sipser or the MIT book). In the general case, I’m a fan learning from textbooks and believe that working things out for yourself without being helped by an instructor makes you stronger, but I’m also convinced that you need feedback from a human when you’re first getting learning how to prove things.
* The lectures follow an old version of the book, which ~350 pages shorter and, crucially, lacks exercises.