Yeah, bearing in mind that the mapping from percentage scores to letter grades or the equivalent is pretty much arbitrary, I much prefer systems where some of the problems presented are really hard and the grade boundaries placed correspondingly lower. It allows for more ambition and more flexibility, and perhaps more importantly it’s just more interesting than a system where you get a perfect score if you don’t screw up each of twenty virtually identical basic exercises. I still have fond memories of a high-school physics class where I once earned an A on a test with a score of 57%. (The median was somewhere in the 30s.)
That presumes it’s real difficulty rather than busywork or pointless procedural stuff, though, which is harder to design and in some fields harder to grade: in mathematics you can grade only on the final answer (with partial credit if you e.g. obviously lost a sign somewhere), but that’s not true for something like e.g. physics lab notebooks.
One problem with this is that the amount of effort you can spend on a set of problems of this sort is nearly unbounded. If the problems are simple enough that a decent understanding of the subject leads you to get them all correct, you will only have to spend as much time as it takes to finish the assignment, and then you’re done. If a decent understanding of the subject only leads you to get 50% correct, then you’ll probably be in a position where you can spend another hour and raise that to 55%, and another hour for 57%, etc. You don’t know (until after the fact) how much you need to get correct to actually get a good grade, so you’re stuck not knowing how much effort is reasonable.
Furthermore, if it’s graded on a curve, this will result in a race to the bottom where everyone spends an extra two hours for that 7% advantage over everyone else and since everyone’s spent it, the overall effect is just that everyone spent an extra two hours for little benefit.
And woe be it if you have two such assignments at the same time. Not only do you have to worry about spending unlimited time because you don’t know when you’re done, it’s going to be very difficult to work on the assignments in order without shifting between one and the other constantly so you don’t spend all your effort on increasing one by 5% when that same effort could have increased the other one by 10%.
the overall effect is just that everyone spent an extra two hours for little benefit.
Woah! I sure hope not! The two or three times I had challenging assignments in school (my school encouraged undergraduates to take graduate classes if interested) they were tremendously valuable. If thinking about difficult problems and solving them has no marginal benefit, I can’t imagine what part of schooling does! (perhaps the diploma mill would be ideal in that scenario? I’m having a hard time simulating this hypothetical student).
It’s not that it has no marginal benefit, it’s that it has diminishing marginal benefit for the effort spent. At some point, you’re better off working on another class’s assignment instead. At some point you’re better off taking leisure time, or even sleeping. If even people with a good understanding are only expected to get 50% correct, how are you supposed to know when you’re better off going to sleep, only knowing that you’ve completed 50% and not knowing whether the extra 5% from sacrificing some sleep is worth it?
The returns diminish when it comes to impact on your grade, yes, and I certainly enjoyed transparency about how the grades I got would be impacted by my work.
The distribution of value for learning, though, goes up with difficulty until it drops to zero (the point at which you cannot solve the puzzle at all). My only point is that we should strongly prefer systems that allow us to soak up all that high-intensity high-value work—modern universities aren’t that for many students, though, but independently reading textbooks could/should be.
Yeah, you don’t want this sort of thing graded on a curve—though the “race to the bottom” issue isn’t substantially worse here than it would be with a more conventional problem set. Curves are generally set by the amount of effort the average student is willing to spend rather than the amount of effort an arbitrarily ambitious student is willing to, meaning that if you’re in a position to be making decisions about how to allocate your limited study time you probably don’t need to be doing so.
The teacher in my example didn’t grade on a curve, he assigned grade boundaries based on his idea of an acceptable level of effort and understanding. He also included estimates of each problem’s difficulty, which was helpful for time management.
As it happens, Art of Problem Solving questions actually fit the bill of being really difficult. Most chapters have about 15 or so “challenge” problems, of which 5 or so are really hard.
The scoring system I used was to (a) give myself double points for answering the really hard challenge problems, and (b) only require a total score of 10 for all the challenge problems combined. So if I got all the regular problems correct + 10 points on the challenge problems (1 point per “standard” challenge problem and 2 points per “hard” challenge problem) then I’d score 100. One side effect of this was that on chapters with a particularly large number of challenge problems it was possible to get something like a 140. But in any case, getting a 100 actually wasn’t that easy.
Yeah, bearing in mind that the mapping from percentage scores to letter grades or the equivalent is pretty much arbitrary, I much prefer systems where some of the problems presented are really hard and the grade boundaries placed correspondingly lower. It allows for more ambition and more flexibility, and perhaps more importantly it’s just more interesting than a system where you get a perfect score if you don’t screw up each of twenty virtually identical basic exercises. I still have fond memories of a high-school physics class where I once earned an A on a test with a score of 57%. (The median was somewhere in the 30s.)
That presumes it’s real difficulty rather than busywork or pointless procedural stuff, though, which is harder to design and in some fields harder to grade: in mathematics you can grade only on the final answer (with partial credit if you e.g. obviously lost a sign somewhere), but that’s not true for something like e.g. physics lab notebooks.
One problem with this is that the amount of effort you can spend on a set of problems of this sort is nearly unbounded. If the problems are simple enough that a decent understanding of the subject leads you to get them all correct, you will only have to spend as much time as it takes to finish the assignment, and then you’re done. If a decent understanding of the subject only leads you to get 50% correct, then you’ll probably be in a position where you can spend another hour and raise that to 55%, and another hour for 57%, etc. You don’t know (until after the fact) how much you need to get correct to actually get a good grade, so you’re stuck not knowing how much effort is reasonable.
Furthermore, if it’s graded on a curve, this will result in a race to the bottom where everyone spends an extra two hours for that 7% advantage over everyone else and since everyone’s spent it, the overall effect is just that everyone spent an extra two hours for little benefit.
And woe be it if you have two such assignments at the same time. Not only do you have to worry about spending unlimited time because you don’t know when you’re done, it’s going to be very difficult to work on the assignments in order without shifting between one and the other constantly so you don’t spend all your effort on increasing one by 5% when that same effort could have increased the other one by 10%.
Woah! I sure hope not! The two or three times I had challenging assignments in school (my school encouraged undergraduates to take graduate classes if interested) they were tremendously valuable. If thinking about difficult problems and solving them has no marginal benefit, I can’t imagine what part of schooling does! (perhaps the diploma mill would be ideal in that scenario? I’m having a hard time simulating this hypothetical student).
It’s not that it has no marginal benefit, it’s that it has diminishing marginal benefit for the effort spent. At some point, you’re better off working on another class’s assignment instead. At some point you’re better off taking leisure time, or even sleeping. If even people with a good understanding are only expected to get 50% correct, how are you supposed to know when you’re better off going to sleep, only knowing that you’ve completed 50% and not knowing whether the extra 5% from sacrificing some sleep is worth it?
The returns diminish when it comes to impact on your grade, yes, and I certainly enjoyed transparency about how the grades I got would be impacted by my work.
The distribution of value for learning, though, goes up with difficulty until it drops to zero (the point at which you cannot solve the puzzle at all). My only point is that we should strongly prefer systems that allow us to soak up all that high-intensity high-value work—modern universities aren’t that for many students, though, but independently reading textbooks could/should be.
Yeah, you don’t want this sort of thing graded on a curve—though the “race to the bottom” issue isn’t substantially worse here than it would be with a more conventional problem set. Curves are generally set by the amount of effort the average student is willing to spend rather than the amount of effort an arbitrarily ambitious student is willing to, meaning that if you’re in a position to be making decisions about how to allocate your limited study time you probably don’t need to be doing so.
The teacher in my example didn’t grade on a curve, he assigned grade boundaries based on his idea of an acceptable level of effort and understanding. He also included estimates of each problem’s difficulty, which was helpful for time management.
As it happens, Art of Problem Solving questions actually fit the bill of being really difficult. Most chapters have about 15 or so “challenge” problems, of which 5 or so are really hard.
The scoring system I used was to (a) give myself double points for answering the really hard challenge problems, and (b) only require a total score of 10 for all the challenge problems combined. So if I got all the regular problems correct + 10 points on the challenge problems (1 point per “standard” challenge problem and 2 points per “hard” challenge problem) then I’d score 100. One side effect of this was that on chapters with a particularly large number of challenge problems it was possible to get something like a 140. But in any case, getting a 100 actually wasn’t that easy.