Exams and Overfitting

When I hear something like “What’s going to be on the exam?”, part of me gets indignant. WHAT?!?! You’re defeating the whole point of the exam! You’re committing the Deadly Sin of Overfitting!

Let me step back and explain my view of exams.

When I take a class, my goal is to learn the material. Exams are a way to answer the question, “How well did I learn the material?”[1]. But exams are only a few hours long, so it’s unfeasible to have questions on all of the material. To deal with this time constraint, an exam takes a random sample of the material and gives me a “statistical” rather than “perfect” answer to the question, “How well did I learn the material?”

If I know in advance what topics will be covered on the exam, and if I then prepare for the exam by learning only those topics, then I am screwing up this whole process. By doing very well on the exam, I get the information, “Congratulations! You learned the material covered on the exam very well.” But who knows how well I learned the material covered in class as a whole? This is a textbook case of overfitting.

To be clear, I don’t necessarily lose respect for someone who asks, “What’s going to be on the exam?”. I understand that different people have different priorities[2], and that’s fine by me. But if you’re taking a class because you truly want to learn the material, in spite of any sacrifices that you might have to make to do so[3], then I’d like to encourage you not to “study for the test”. I’d like to encourage you not to overfit.

[1] When I say “learned”, I mean in the “Feynman” sense, not in the “teacher’s password” sense. I believe that a necessary (but not sufficient) condition for an exam to check for this kind of learning is to have problems that I’ve never seen before.

[2] Someone might care much more about getting into medical school than, say, mastering classical mechanics. I respect that choice, and I acknowledge that someone might be in a system where getting a good grade in physics is required for getting into medical school, even though mastering classical mechanics isn’t required for becoming a good doctor.

[3] There were a few terms when I felt like I did a really good job of learning the material (conveniently, I also got really good grades during these terms). But for these terms, one (or both) of the following would happen:

  • I would take a huge hit in social status, because I was taking barely more than the minimum courseload. At my university, there was a lot of social pressure to always take the maximum courseload (or petition to exceed the maximum courseload), and still participate in lots of extracurricular activities.

  • My girlfriend at the time would break up with me because of all the time I was spending on my coursework (and not with her).