Unironically eating the french omelette is bad. But it’s a great ‘chef test’ precisely because it sucks and is tricky to get to mediocre.
The Thai style omelette is obviously a superior thing to eat, but by virtue of being such a good idea it will tend to apologize for the imperfections of the chef making it. If the Thai omelette is a 20x multiplier on execution, then it’s more than 20x harder to tell the difference between good at execution and great at execution. Your ability to judge quality hits a ceiling at some point and you get a bunch of candidates where you go ‘wow, this guy’s really good’, and then a few months later the flaws become more noticeable. The french omelette, by virtue of being a terrible idea, takes you to a scale where the gradient of chef quality is easier to judge.
I think you’re generally right that people tend to miss the point that you should not put great tests on pedestals and that it’s silly to order french omelettes at restaurants. But proxy variables do have their place. I guess my main gripe is the rest of the post doesn’t seem to follow from how you constructed the omelette story.
Unironically eating the french omelette is bad… and that it’s silly to order french omelettes at restaurants
Huh, really? I didn’t realize that, I thought it was considered to be both a good chefs test and a good dish. I just skimmed through the Serious Eats post again though and am not actually seeing a claim that it is such a good dish though, so maybe you’re right. I’ve never actually had one myself.
but by virtue of being such a good idea it will tend to apologize for the imperfections of the chef making it. If the Thai omelette is a 20x multiplier on execution, then it’s more than 20x harder to tell the difference between good at execution and great at execution.
I don’t think that’s necessarily true. It’s possible that a dish has a large spread: it can be, say, 20x better if executed well and 20x worse if executed poorly. How good a dish is (when executed reasonably well) and how forgiving it is are two separate things. But perhaps they’re correlated.
I think you’re generally right that people tend to miss the point that you should not put great tests on pedestals and that it’s silly to order french omelettes at restaurants. But proxy variables do have their place. I guess my main gripe is the rest of the post doesn’t seem to follow from how you constructed the omelette story.
The main idea isn’t really about tests and proxy variables. To use a different analogy, it’s about the explore-exploit tradeoff. Sometimes continuing to exploit is unwise and you’d be much better off exploring entirely different “dishes”.
Unironically eating the french omelette is bad. But it’s a great ‘chef test’ precisely because it sucks and is tricky to get to mediocre.
The Thai style omelette is obviously a superior thing to eat, but by virtue of being such a good idea it will tend to apologize for the imperfections of the chef making it. If the Thai omelette is a 20x multiplier on execution, then it’s more than 20x harder to tell the difference between good at execution and great at execution. Your ability to judge quality hits a ceiling at some point and you get a bunch of candidates where you go ‘wow, this guy’s really good’, and then a few months later the flaws become more noticeable. The french omelette, by virtue of being a terrible idea, takes you to a scale where the gradient of chef quality is easier to judge.
I think you’re generally right that people tend to miss the point that you should not put great tests on pedestals and that it’s silly to order french omelettes at restaurants. But proxy variables do have their place. I guess my main gripe is the rest of the post doesn’t seem to follow from how you constructed the omelette story.
Huh, really? I didn’t realize that, I thought it was considered to be both a good chefs test and a good dish. I just skimmed through the Serious Eats post again though and am not actually seeing a claim that it is such a good dish though, so maybe you’re right. I’ve never actually had one myself.
I don’t think that’s necessarily true. It’s possible that a dish has a large spread: it can be, say, 20x better if executed well and 20x worse if executed poorly. How good a dish is (when executed reasonably well) and how forgiving it is are two separate things. But perhaps they’re correlated.
The main idea isn’t really about tests and proxy variables. To use a different analogy, it’s about the explore-exploit tradeoff. Sometimes continuing to exploit is unwise and you’d be much better off exploring entirely different “dishes”.