a famous resulf in psychology is that if you tell a class of photography students to produce one extremely high quality photo, they actually produce worse photos than another class told only to produce a lot of decent photos, because in making lots of decent photos you learn way more about photography through trial and error, whereas the high quality photo group tends to spend irrationally much time making each try as perfect as possible.
does this effect actually replicate, especially across domains?
On the other hand, I’ve heard the opposite advice for novice chess players (basically playing blitz without a good foundation destroys your ability to improve).
I interpreted the saying as “play a lot of games without overthinking”, not “play blitz”.
I’ve taught a few people how to play Go, and for the first 20–50 games, players are basically clueless about what they’re even supposed to be doing—they can place stones anywhere, and they have no notion of why they should place a stone in any particular place.* I think it’s better to develop basic intuition by playing rather than by studying what you’re supposed to do.
*I think chess and Go are different in that respect. In chess, it quickly becomes obvious that the correct thing to do is to develop your pieces.
I think in Go playing many quick games is generally better for novices than playing slow games for “initializing your neural network” so to speak. Among others, developing an intuitve and holistic sense for which shapes lead to victory (sometimes it’s referred to as “beautiful” or “elegant”) matters a lot, even at low levels.
Whereas in chess novice play is almost entirely dominated by tactics, which means playing lots of quick games often teaches you bad patterns and shortcuts.
TLDR: Yes there are a few experiments that have found this result, but only in very short-term studies, so I’m not very confident that it’s a real effect. (I’d say the studies are maybe a 1.5:1 update in favor. My common-sense prior is 66%, so that puts my posterior at 75%.)
Just curious: You made the strong claim that it was a famous result, didn’t you know whether there were actual experiments before you made that claim? I’m just wondering what level of rigour to expect with these quick takes.
There’s a nice book called the success equation that makes a point similar to the following paper about skill vs luck based domains where a skill domain is something where you’re able to get repeat successes without much variance and a luck domain where things are more random. There’s a sort of implicit model in the photography thing where you have higher variance.
The advice is to take the most amount of shots in the luck based domain as that is all that matters there but what about the skill based domain?
Well, there you want to (approximately) maximise your learning so we might say that the actions we should pick should be the ones that are dependent on us learning something. A naive approach is something like max(n*p) where p is the probability of learning and n is the amount of opportunities you have to learn. Sometimes you don’t want to do this because the skill lies in highly specific skills which means you want to spend a longer time on one attempt so that you get this really right. This then essentially boils down to a learning problem where priors are inherited through previous people often (e.g different sports have different training regimes).
But what do you do if you don’t know the extent to which something is luck vs skill?
I think the answer basically becomes to just do the photography thing again and see what the variance becomes over time, treat your unknown domain is as if it were a skill domain and see if it lowers the variance of it. If it does it probably is, if it isn’t then you’ve already done the strategy of taking many shots which means you’ve done the right thing.
I have tried this out in practice over the last year or so and I’ve concluded that mentorship is probably a more skill based domain than I originally thought and that writing is more luck based than I thought (I might just not have enough data on this one though.)
Clarification after chatting with claude: It’s the conditional probability of the mean moving that matters p(mean|action). Secondly it is about feedback signal clarity for learning, can you move fast and get a clean signal? (e.g blitz chess) Or does moving fast make it so that you can’t learn (e.g trying to learn to code by using claude to vibe-code apps that you don’t even look at the code of).
There’s also some annoying stuff around how getting high variance in the results doesn’t mean the domain isn’t skill based since it is only the conditional on the mean moving that matters. So the writing thing might actually not be true in terms of being a non-skilled domain, it is more that it is inherently high variance in the feedback domain.
At least in mechanical engineering many smaller scope, approachable projects will teach much more than working on a one off, ambitious project: the latter often stay paper projects or fail for want of basic experience.
I would say that it’s more about time horizon. Just randomly trying things leads to basically just doing gradient descent until you hit a local optimum, while a more intentional approach looks worse at first but then you can eventually hit somewhere even better. So if your goal is “best photos by end of short photography class,” then just trying things out is the best approach. If your goal is “be an expert chess player” or “be an expert photographer” or anything like that, I think you need to be more intentional.
To use a bit of an analogy, evolution (basically massive, repeated trial and error) produced the cheetah, which can go at 65mph, and that’s genuinely impressive, but intentionally designing something to go quickly (e.g. an airplane) can go orders of magnitude faster.
I’ve seen “spam volume” in dating advice, putting out a lot of steam games to appeal to a certain audience (one asset-flippy kind of dev was mentioned as doing that on a YouTube video), and how LLM responses are easy to ask for in a way where you can ask multiple LLM’s and choose what you like (either to confirm accuracy or people who are cycling through a lot of prototype designs). I’m not sure it applies to all domains but the more general “do more = good” seems to obviously replicate.
This would be interesting to me as well to test, but in some of the aspects I am not sure how you’d compare, for example paying unskilled gamedevs to make many small games instead of a big one
I mean, you’d get exactly the same results even if the pupils learned absolutely nothing and the quality of the photo has some kind of stable distribution. To control if people did or didn’t learn anything, one would at least need to perform this as an actual experiment and grade the 3 next photos after the learning phase was done. The initial story seems to be only a parable as per Claude of one commenter below and my ChatGPT instance. My personal opinion is that at the beginner level, doing some amount of attempts does help your brain make sense of what’s happening; afterwards, it’s the feedback loops which are key. And specifically in this example, from some point at the skill curve, to learn to make great shots, one would need to practice making great shots with a teacher.
i’ve found this replicates in writing as a way of preventing writer’s block. initially, I thought the issue here is that the subjectiveness of the goal (i.e. producing a high quality photo) creates inertia, but even on tasks that have a clear success criterion (i.e. writing the fastest kernel), this happens.
we just aren’t too good at reaching the optimal point immediately (and its often not known what this point looks like), we just do noisy gradient steps and make local updates in real life
a famous resulf in psychology is that if you tell a class of photography students to produce one extremely high quality photo, they actually produce worse photos than another class told only to produce a lot of decent photos, because in making lots of decent photos you learn way more about photography through trial and error, whereas the high quality photo group tends to spend irrationally much time making each try as perfect as possible.
does this effect actually replicate, especially across domains?
A piece of standard advice for new Go players is to lose your first 50 games as quickly as possible rather than attempting to study theory before you’ve actually played a bunch.
On the other hand, I’ve heard the opposite advice for novice chess players (basically playing blitz without a good foundation destroys your ability to improve).
Arjun Panickserry reviews a book that’s vaguely “quantity over quality” for chess
I interpreted the saying as “play a lot of games without overthinking”, not “play blitz”.
I’ve taught a few people how to play Go, and for the first 20–50 games, players are basically clueless about what they’re even supposed to be doing—they can place stones anywhere, and they have no notion of why they should place a stone in any particular place.* I think it’s better to develop basic intuition by playing rather than by studying what you’re supposed to do.
*I think chess and Go are different in that respect. In chess, it quickly becomes obvious that the correct thing to do is to develop your pieces.
I think in Go playing many quick games is generally better for novices than playing slow games for “initializing your neural network” so to speak. Among others, developing an intuitve and holistic sense for which shapes lead to victory (sometimes it’s referred to as “beautiful” or “elegant”) matters a lot, even at low levels.
Whereas in chess novice play is almost entirely dominated by tactics, which means playing lots of quick games often teaches you bad patterns and shortcuts.
This is the sort of question LLMs are good at answering: https://claude.ai/share/06e3e68f-933e-4e2f-be11-83f036e22947
TLDR: Yes there are a few experiments that have found this result, but only in very short-term studies, so I’m not very confident that it’s a real effect. (I’d say the studies are maybe a 1.5:1 update in favor. My common-sense prior is 66%, so that puts my posterior at 75%.)
I don’t think it was a study or result at all, just a parable by someone to assert the solution.
https://excellentjourney.net/2015/03/04/art-fear-the-ceramics-class-and-quantity-before-quality/
has someone run the experiment after the assertion?
Just curious: You made the strong claim that it was a famous result, didn’t you know whether there were actual experiments before you made that claim? I’m just wondering what level of rigour to expect with these quick takes.
Even if they don’t learn anything, they’re still getting more opportunities for something to work out well.
i would love to fund a microgrant for someone to actually run an experiment like this
There’s a nice book called the success equation that makes a point similar to the following paper about skill vs luck based domains where a skill domain is something where you’re able to get repeat successes without much variance and a luck domain where things are more random. There’s a sort of implicit model in the photography thing where you have higher variance.
The advice is to take the most amount of shots in the luck based domain as that is all that matters there but what about the skill based domain?
Well, there you want to (approximately) maximise your learning so we might say that the actions we should pick should be the ones that are dependent on us learning something. A naive approach is something like max(n*p) where p is the probability of learning and n is the amount of opportunities you have to learn. Sometimes you don’t want to do this because the skill lies in highly specific skills which means you want to spend a longer time on one attempt so that you get this really right. This then essentially boils down to a learning problem where priors are inherited through previous people often (e.g different sports have different training regimes).
But what do you do if you don’t know the extent to which something is luck vs skill?
I think the answer basically becomes to just do the photography thing again and see what the variance becomes over time, treat your unknown domain is as if it were a skill domain and see if it lowers the variance of it. If it does it probably is, if it isn’t then you’ve already done the strategy of taking many shots which means you’ve done the right thing.
I have tried this out in practice over the last year or so and I’ve concluded that mentorship is probably a more skill based domain than I originally thought and that writing is more luck based than I thought (I might just not have enough data on this one though.)
Clarification after chatting with claude: It’s the conditional probability of the mean moving that matters p(mean|action). Secondly it is about feedback signal clarity for learning, can you move fast and get a clean signal? (e.g blitz chess) Or does moving fast make it so that you can’t learn (e.g trying to learn to code by using claude to vibe-code apps that you don’t even look at the code of).
There’s also some annoying stuff around how getting high variance in the results doesn’t mean the domain isn’t skill based since it is only the conditional on the mean moving that matters. So the writing thing might actually not be true in terms of being a non-skilled domain, it is more that it is inherently high variance in the feedback domain.
This is extremely common conventional wisdom in writing. No idea how accurate it is.
Wasn’t the initial example around pottery?
At least in mechanical engineering many smaller scope, approachable projects will teach much more than working on a one off, ambitious project: the latter often stay paper projects or fail for want of basic experience.
Yes, recommend John Cleese’s Creativity talk. See also research on brainstorming.
I would say that it’s more about time horizon. Just randomly trying things leads to basically just doing gradient descent until you hit a local optimum, while a more intentional approach looks worse at first but then you can eventually hit somewhere even better. So if your goal is “best photos by end of short photography class,” then just trying things out is the best approach. If your goal is “be an expert chess player” or “be an expert photographer” or anything like that, I think you need to be more intentional.
To use a bit of an analogy, evolution (basically massive, repeated trial and error) produced the cheetah, which can go at 65mph, and that’s genuinely impressive, but intentionally designing something to go quickly (e.g. an airplane) can go orders of magnitude faster.
I’ve seen “spam volume” in dating advice, putting out a lot of steam games to appeal to a certain audience (one asset-flippy kind of dev was mentioned as doing that on a YouTube video), and how LLM responses are easy to ask for in a way where you can ask multiple LLM’s and choose what you like (either to confirm accuracy or people who are cycling through a lot of prototype designs). I’m not sure it applies to all domains but the more general “do more = good” seems to obviously replicate.
This would be interesting to me as well to test, but in some of the aspects I am not sure how you’d compare, for example paying unskilled gamedevs to make many small games instead of a big one
I mean, you’d get exactly the same results even if the pupils learned absolutely nothing and the quality of the photo has some kind of stable distribution. To control if people did or didn’t learn anything, one would at least need to perform this as an actual experiment and grade the 3 next photos after the learning phase was done. The initial story seems to be only a parable as per Claude of one commenter below and my ChatGPT instance.
My personal opinion is that at the beginner level, doing some amount of attempts does help your brain make sense of what’s happening; afterwards, it’s the feedback loops which are key. And specifically in this example, from some point at the skill curve, to learn to make great shots, one would need to practice making great shots with a teacher.
i’ve found this replicates in writing as a way of preventing writer’s block. initially, I thought the issue here is that the subjectiveness of the goal (i.e. producing a high quality photo) creates inertia, but even on tasks that have a clear success criterion (i.e. writing the fastest kernel), this happens.
we just aren’t too good at reaching the optimal point immediately (and its often not known what this point looks like), we just do noisy gradient steps and make local updates in real life