Exams and Overfitting

When I hear some­thing like “What’s go­ing to be on the exam?”, part of me gets in­dig­nant. WHAT?!?! You’re defeat­ing the whole point of the exam! You’re com­mit­ting the Deadly Sin of Overfit­ting!

Let me step back and ex­plain my view of ex­ams.

When I take a class, my goal is to learn the ma­te­rial. Ex­ams are a way to an­swer the ques­tion, “How well did I learn the ma­te­rial?”[1]. But ex­ams are only a few hours long, so it’s un­fea­si­ble to have ques­tions on all of the ma­te­rial. To deal with this time con­straint, an exam takes a ran­dom sam­ple of the ma­te­rial and gives me a “statis­ti­cal” rather than “perfect” an­swer to the ques­tion, “How well did I learn the ma­te­rial?”

If I know in ad­vance what top­ics will be cov­ered on the exam, and if I then pre­pare for the exam by learn­ing only those top­ics, then I am screw­ing up this whole pro­cess. By do­ing very well on the exam, I get the in­for­ma­tion, “Con­grat­u­la­tions! You learned the ma­te­rial cov­ered on the exam very well.” But who knows how well I learned the ma­te­rial cov­ered in class as a whole? This is a text­book case of overfit­ting.

To be clear, I don’t nec­es­sar­ily lose re­spect for some­one who asks, “What’s go­ing to be on the exam?”. I un­der­stand that differ­ent peo­ple have differ­ent pri­ori­ties[2], and that’s fine by me. But if you’re tak­ing a class be­cause you truly want to learn the ma­te­rial, in spite of any sac­ri­fices that you might have to make to do so[3], then I’d like to en­courage you not to “study for the test”. I’d like to en­courage you not to overfit.

[1] When I say “learned”, I mean in the “Feyn­man” sense, not in the “teacher’s pass­word” sense. I be­lieve that a nec­es­sary (but not suffi­cient) con­di­tion for an exam to check for this kind of learn­ing is to have prob­lems that I’ve never seen be­fore.

[2] Some­one might care much more about get­ting into med­i­cal school than, say, mas­ter­ing clas­si­cal me­chan­ics. I re­spect that choice, and I ac­knowl­edge that some­one might be in a sys­tem where get­ting a good grade in physics is re­quired for get­ting into med­i­cal school, even though mas­ter­ing clas­si­cal me­chan­ics isn’t re­quired for be­com­ing a good doc­tor.

[3] There were a few terms when I felt like I did a re­ally good job of learn­ing the ma­te­rial (con­ve­niently, I also got re­ally good grades dur­ing these terms). But for these terms, one (or both) of the fol­low­ing would hap­pen:

  • I would take a huge hit in so­cial sta­tus, be­cause I was tak­ing barely more than the min­i­mum courseload. At my uni­ver­sity, there was a lot of so­cial pres­sure to always take the max­i­mum courseload (or pe­ti­tion to ex­ceed the max­i­mum courseload), and still par­ti­ci­pate in lots of ex­tracur­ricu­lar ac­tivi­ties.

  • My girlfriend at the time would break up with me be­cause of all the time I was spend­ing on my course­work (and not with her).