[Link] Quantity Always Trumps Quality
http://www.codinghorror.com/blog/2008/08/quantity-always-trumps-quality.html
The ceramics teacher announced on opening day that he was dividing the class into two groups. All those on the left side of the studio, he said, would be graded solely on the quantity of work they produced, all those on the right solely on its quality. His procedure was simple: on the final day of class he would bring in his bathroom scales and weigh the work of the “quantity” group: fifty pound of pots rated an “A”, forty pounds a “B”, and so on. Those being graded on “quality”, however, needed to produce only one pot—albeit a perfect one—to get an “A”.
Well, came grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity. It seems that while the “quantity” group was busily churning out piles of work—and learning from their mistakes—the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.
For some reason it just seems we in particular could learn something from this anecdote.
Iterate more. The practice effect is your friend as is mining out positive outliers in really huge sets. I wanted to also mention something about using going meta as a way to procrastinate but I feared I would summon a Newsome.
Edit: This has been mentioned before. I think it is good to remind people of it.
Not only has it been mentioned before, last time it came up I searched and failed to find corroboration of the claim that it actually happened. Since applying a deliberately inconsistent grading rubric is not something professors are normally allowed to do, I strongly suspect that the anecdote is fictional.
It is therefore best to assume this is a parable.
I think this might be a ceramics thing in particular. Throwing pots is hugely about practice effects and muscle memory and feeling the clay thin and unbalance before it goes pear-shaped. There are probably other fields to which this pattern applies, but let’s not assume it’s a general rule.
For example, I bet there are fields where doing a lot of something without caring about results quickly plateaus your quality and then drops it as you start cutting corners, but concentrating on each individual piece lets you ramp up quality over a few iterations without that deterioration.
I would expect iteration to be more useful when clear and immediate feedback is available.
Seriously. Dang clay. Still hella fun though.
Yeah, I miss ceramics. A potter’s wheel tops the list of Things I Would Do With Lots Of Empty Rooms To Fill.
Edit: Why do I keep being wistful about things and not Googling? Two minutes of looking found me a potter’s studio with day passes and firing services within walking distance.
Umm, “always”? Generalizing from one example?
Also seems like a false dichotomy. I bet that a group that was encouraged to think a bit before making each new pot would have done better yet.
Testing and finding one thing superior to another isn’t a ‘false dichotomy’. It is merely a comparison between two things. It requires that someone say or strongly imply that there are not any other alternatives to the two for it to be any kind of dichotomy. Nobody seems to be doing that.
I didn’t choose the original title.
Not only has it been mentioned before, last time it came up I searched and failed to find corroboration of the claim that it actually happened. Since applying a deliberately inconsistent grading rubric is not something professors are normally allowed to do, I strongly suspect that the anecdote is fictional.
I’ve edited the OP to include this so it doesn’t mislead readers.
I see your anecdote, and I raise it and the evidence hierarchy with Ericsson’s correlational research on “deliberate practice”.
Did Ericsson study types of practice for beginners to good amateurs? The Cambridge Handbook of Expertise and Expert Performance (I believe this is your copy?) wouldn’t cite that research. Ericsson’s papers apparently all have to do with experts. There’s one cross-performance-levels paper, about college students, which finds that studying in quiet environments and going to class help more than studying longer, which is consistent with deliberate practice models but also with many others.
I’m not sure he personally studied them, or just discussed the previous work. eg. from his ’93 paper discussing the violinists:
I see your bizarre, not-quite-comprehensible retort, assume from your follow ups that you are contradicting the presented claim and raise you a “What the heck? Actually go read Ericsson et al. before you engage in this kind of petty condescension.”
This finding is entirely in keeping with the literature and this outcome is exactly what one would expect to observe in such a situation. Ericsson and other ‘real researchers’ have done extensive related studies on what kind of goal produces the best outcomes depending on level of expertise. The ‘quality’ goal fares consistently poorly for the beginner, across disciplines. (It is better to switch to aiming for quality once already at a reasonably high level.)
‘Deliberate practice!!!’ is an important finding, but it isn’t the only one out there. You have abused the appeal to the authority of Ericsson.
Out there in hypothesis land, I’m wondering whether beginners need something close to play—they can’t do directed practice yet because they don’t know what goals to aim for, but they do need to acquire a large quantity of tacit knowledge.
I have read a number of Ericsson’s papers, and most of the Cambridge Handbook (you may remember that it was my pirated edition that LWers used for a while); where do they say an equal number of hours of indiscriminate practice is best for beginners as compared to the equivalent effort devoted differently?
Certainly I agree that beginners benefit most from lots of time spent practicing (total time practicing was a powerful predictor in the studies), but the anecdote in OP was not about two groups of students, one who studied 1 hour a week and the other studied 1 hour a day...
I am not, nor have I ever advocated indiscriminate practice as a preferred form of training.
This isn’t about time spent. This is about what goal the participants have while doing the activity, with all else being equal. ‘Quality’ is, empirically, a terrible goal for beginners to be given.
How does the OP’s claim that the pottery students were better off producing as much as possible by weight not constitute advocacy of indiscriminate practice?
Are you going to provide any cites?
I can’t parse your sentence, so I have no idea what you are saying, but two things that may be relevant:
This is a randomized controlled experiment, not an anecdote.
This experiment shows benefit from arbitrary practice, not deliberate practice.
You don’t know that it was an actual experiment of any kind, so it remains an anecdote: this story is now at least at third-hand—the source to the book to Atwood (to Konkvistador). Some searches turn up nothing beyond Atwood, and the book is not usefully available online (Google Books at least confirms some sort of passage like that is in it, but provides no context—like perhaps any mention by the author that it’s just a story he was once told or something like that).
This claims benefit from arbitrary practice; real research by real researchers where one can give real citations shows an overall effect that is opposite: that people who practice a lot can quickly plateau no matter how many decades they spend on it, and deliberate practice is necessary to go beyond that.
Well, it might even be a parable, which is even worse than an anecdote!
If the point of your original comment was to say that this contradicts Ericsson’s work, I suggest you rewrite it.
There are lots of reasons to be skeptical of this claim, but calling it an “anecdote” evokes all the wrong ones. Frankly, I can only describe this usage as deceit. I blame the hierarchy of evidence.
Furthermore, if that was the point he should retract it in shame.
Why do you care that it is third-hand? Do you think something was lost in the transmission from Atwood to Konvistador? It is easy to check that nothing was lost there. The fact that it is third-hand is evidence of memetic stability, but tracking down the book will not change that. (and I could level the same charge at Ericsson)
In fact, I did look at the book and I assert (sadly) that nothing was lost in Kevin Kelly’s transcription. It is the first paragraph of the section and moves on to drawing conclusions.
Sourcing is key because every step introduces error. And you misunderstand: Atwood to Konkvistador would be a fourth-hand, if one wanted to include it. Count the steps.
So then, it’s essentially worthless. There is no citation, no context, nothing to situate it in any time, place, country or year besides ‘the 20th century’: we can’t even tell how many steps we are removed from the origin since you say there was no context like “many years ago, an old friend of mine was taking a pottery class”.
Just another persuasive parable floating around.
I’ll stick with Ericsson’s research, thanks. Mindless practice is not useful; deliberate practice is useful.
Where does Ericsson say this?
He doesn’t say that explicitly that I can recall; it’s just the theme running through his research, background research, and the theories.
Thanks for teaching me a lesson on second-hand sources.
I didn’t see this parable extolling the virtue of mindless practice rather than the virtue of doing huge amounts of work if you really want to create remarkable outliers.
What ensures “memetic stability”, Heisenberg compensators? Or just the fact that it was passed along?
Ah, the zeugma—always a classy figure of speech, but sometimes a risky one. Victor Hugo was good at them.
Part of gwern’s statement is that it’s not actually certain that this really happened. It might not even be an anecdote, but just a parable!
It’s been noted before, but it’s a great read.
Your post in particular is a very nice reminder to me because I’m currently in the process of procrastinating writing. I sat down at my computer with express intention to write, I thought “Oh god what if nobody will like my story”, then I came to LessWrong to intentionally procrastinate because I was afraid I wouldn’t get it right the first time. I’d better go back and churn out some words. Thanks for the reminder, sir.
I will edit the OP to link that discussion.
Upvoted for the parable, but I don’t like the title of the post.
Expect to regress to the mean. As encouraging as large positive outliers are, also don’t be discouraged by large negative outliers.
This one seems obvious but is difficult to remember in the moment.
I don’t understand this, could you explain, please?
Say you throw a single fair die, what is the probability you get a 6 on your roll?
Now say your throw the same die ten times recording the result of each roll. What is the probability that you will have at least one 6 recorded on your sheet of paper?
If that six is say a great work of art, people will usually not care about all the other 1-5s you accumulated in the search for your six.
If however someone is grading you on both your positive and negative outliers, perhaps a game where your score is equal to the average of all dice throws, you won’t be better off with a set of a 100 die throws than with a set of just one die roll. Indeed if you had a tournament with that game, all the champions (and all the worst losers) would be people who threw the die just once, because 100 dice rolls with an average result of 6 are far more improbable than a set of one die roll with an average result of 6.
This sounds like an awfully easy system to game in any case. One can simply make a single pot out of over fifty pounds of clay.
Of course, it probably won’t bake very well, even putting aside the fact that it’ll probably be a poor piece of handiwork, but quality is explicitly a non-issue.
I don’t think fifty pounds of clay would fit on a normal potter’s wheel. One pound of clay is enough to make a mug—a big mug, if you know what you’re doing; one with really thick walls if you don’t. But you could make ten five-pound bowls. (I say bowls because they’re easy to get by accident if you’re trying for cylinders.)
I suspected this was probably the case, but then, the pot doesn’t necessarily have to be made on a potter’s wheel. There are pre-wheel methods for making pottery, but they’re mostly pretty slow, so I would probably experiment with the method of jumping up and down on a fifty pound mass of clay to create a depression, and shaping the clay around that by hand.
I suspect that if that grading procedure were applied in real life, you’d get some wags like me for whom willful perversity would trump efficiency, who would insist on making a single pot even if it turned out to be more difficult than making a handful of really heavy ones, and also a bunch of procrastinators who made all their pots in a rush at the last minute and thus failed to benefit much from practice, turning out a bunch of crappy pots.
If the professor wanted to encourage students to get the benefits of practice, they’d probably get better results asking the students to bring in a new pot every class.
Surely, one of the things I learn here: you have to do right on the first.
But, in the history of AI, most architectures are based on the ultimate insight. Simplified, and wrong. The dichotomy is useful to people who think too much without scientific constrains, I suppose.
Video (5 minutes) by Victor Wooten about learning one’s first language by jamming with the professionals.