Joint Distributions and the Slow Spread of Good Ideas
A few years ago a well-known economist named David Romer published a paper in a top economics journal* arguing that professional football teams don’t “go for it” nearly often enough on fourth down. The question, of course, is how this can persist in equilibrium. If Romer is correct, wouldn’t teams have a strong incentive to change their strategies? Of course it’s possible that he is correct, but that no one ever knew it before the paper was published. But then would the fact that the recommendation has not been widely adopted** constitute strong evidence that he is not correct? The paper points out two possible reasons why not. First, the objective function of the decision-makers may not be to maximize the probability of winning the game. Second and more relevant for our purposes, there may be some biases at work. The key point is this quote from the article (page 362):
“Many skills are more important to running a football team than a command of mathematical and statistical tools. And it would hardly be obvious to someone without knowledge of those tools that they could have any significant value in football.”
Romer’s point is that what’s relevant is the joint distribution of attributes in the pool of potential football coaches (or other decision-makers). Even in something like professional football where there is a very strong incentive to get better results, it may take a long time for coaches who are willing/able to adopt a good new idea to out-compete and displace those who continue to use the bad old idea if there are very few potential coaches who both have the conventional talents and understand the new idea.
I think Romer is right about this, and his point is the main take-away point of this post. But I don’t think the main “joint distribution” problem is a paucity of would-be coaches who understand both conventional football stuff and math: math talent can be hired to work under a head coach who doesn’t understand it, just like medical talent is. Rather, it needs to be the case that the joint distribution is unfavorable and that this can’t be gotten around by just adding math talent as necessary. Maybe the problem is a paucity of potential coaches who have both conventional coaching skills and also the attitude that nerds are to be listened to rather than beaten up. This may explain why baseball seems to have been much more accepting of statistical analysis than has football.
The point is this: an unfavorable joint distribution of attributes in the pool of potential decision-makers can greatly retard the adoption of good ideas, even when incentives to adopt are very strong, which means that the fact of non-adoption is not decisive evidence that an idea is bad. But for this to be true, there must be some reason why the people with the principal attribute cannot simply seek and incorporate the advice of the people with the secondary attribute. This will often be because the very acculturation process that produced the people who have the principal attribute creates some barrier to them making use of the secondary one.
”Do Firms Maximize? Evidence from Professional Football.” by David Romer, Journal of Political Economy (2006). It is a sad commentary on the state of the economics profession that some journal editor seems to have made him change the title from the much cooler: “It’s Fourth Down and What Does the Bellman Equation Say? A Dynamic-Programming Analysis of Football Strategy.”
**I think this is a true statement, but I could be wrong. Please correct me if I am.
***Another possibility is that the problem is not the coaches, but the fans. A coach who sticks with the conventional strategy is protected by a “nobody ever got fired for buying IBM” attitude, whereas a coach who does something unconventional (but probabilistically correct) runs the risk of getting fired if it doesn’t work out. This just pushes the irrationality from the coaches to the fans, but that might be more plausible: they have access to less resources to figure out what is and is not a good idea, and have much less of an incentive to try to get it right. Then the problem would be a paucity of fans who have the attribute “really care about football” and “understand and are willing to support good ideas, even from nerds.”
If I remember correctly, there was at least one team that responded to that paper by changing its tactics, and did very well for itself: the Patriots.
ETA: an article that mentions it: http://www.nytimes.com/2004/12/12/magazine/12SABER.html
I didn’t know about that. Thanks! Belichick also seems to be something of a cheater cheater pumpkin eater, so I guess he’s just willing to take help from wherever he can get it.
In the community of sports statistical analysis, the most-accepted hypothesis is that coaches are reluctant to try new strategies for rational reasons. If the new strategy succeeds, they get a bit of utility, but if the new strategy fails, they get fired—and so lose a lot more utility.
Being a maverick has a negative expectation for the coach, even though it might have a positive expectation for the team.
This hypothesis makes a lot more sense to me than assuming that coaches are unaware of the result.
Yes. And since being a maverick has a similar negative expectation for most working people, it seems well-placed to explain the slow spread of good ideas more generally as well.
But there has to be some irrationality in there somewhere. That is, if you place your job at risk by not buying IBM even when it is not the best choice, it has to be either that the person who gets to decide whether or not to fire you has wrong ideas, or that the people that person needs to satisfy have wrong ideas. So in the football example, if it’s not the coaches, it has to be someone else, most likely the fans.
It seems like it would be rational to fire someone who diverged from traditional methods and then lost. Even though there’s a correlation between running and winning, that doesn’t show causation, and the coach might just be barking up the wrong tree. And then he’s “someone who bet your money on some crazy idea”, who doesn’t seem like the sort of person you’d want working for you.
Of course, on Monday morning, we can easily tell that the coach should not be fired for this particular plan.
It’s true that it can be rational to fire someone who doesn’t buy IBM if the decision not to buy IBM constitutes evidence that worker is of low quality. But I don’t think that’s the case here. I think that if a football coach got fired for going for it on fourth down more often than most, he would have been fired because of that decision itself, not because going on fourth down represents a signal that he’s just generally a bad coach.
It’s also required that there be some reason why people with the secondary skill can’t train themselves in the primary attribute and displace the original set. This approach is essential to many entrepreneurs, for instance.
This does not, of course, detract from your argument.
I think that’s right. More broadly, the “unfavorable joint distribution of attributes retards adoption of good ideas” phenomenon doesn’t apply when the limitations imposed by the joint distribution can be easily gotten around one way or the other. That’s why I suspect it is most at play when one of the scarce attributes is actually the willingness to entertain doing the things that would be necessary to escape the constraints imposed by the other scarce attributes.
Yep, economists call this barriers to entry.
See also Hard Facts, about how much is known about business management (frex, that designing a non-destructive incentive scheme is only possible in rare cases) and how frequently it’s ignored.
This reminds me of the TV series House. Dr. House, the maverick, is always coming up with courses of action that are rationally correct, and Dr. Cuddy, his boss, is always striking them down as unethical or something. I wonder how many members of the general population side with Cuddy.
But House’s “rationally correct” courses of action are only eventually proven so with hindsight. They often involve harsh experiments that directly endanger the patient, and each episode generally includes at least 2 dangerous false diagnoses for dramatic effect.
In real life, House would by now have killed an unacceptable proportion of his patients and got himself fired and sued. I love the series, so I don’t mind suspending disbelief about that.
You should be able to make a correct but unexpected decision without being branded a maverick—sometimes. To be a successful maverick your correct decisions have to be unexpected most of the time. But then those around you who supposedly know the subject would have to be badly miscalibrated...
Supposedly this is House’s gift; he is able to figure out cases that are mysterious to other doctors. So it makes sense that, with respect to the cases he sees, other doctors’ calibration would be poor. (But then you wonder why they don’t defer to him more...)
Because they hate being shown up. Plenty of real life precedent for that.
The specific example I had in mind was in the episode “Son of Coma Guy”, in which a father, who is going to fall into a coma for the rest of his life, can save his son’s life with a heart transplant, and decides he’d rather die to save his son’s life than live in a coma while his son dies. House would like to proceed with the transplant, but Cuddy refuses.
That seems closer to shut up and multiply.
I didn’t know about that. Thanks!
See also Hard Facts, which has plenty about what’s known about management (frex, that non-destructive incentive plans are very hard to design) and how much it’s ignored.
I think your point is a good one, but I’m not sure about the example.
It seems just as likely to be that the economist in question doesn’t understand elements of football enough for his advice to be good.
I guess I should read the paper. But just because more often than not 4th down attempts succeed, does not mean they should be attempted more often. The gain of a successful 4th down attempt—four more downs—is much smaller than the risk—giving the opponent’s psychological motivation and good field position. So coaches should only attempt 4th downs when the odds strongly favor success, and a high level of success indicates that is what coaches do.
Now I presume Romer attempted to take things like this into account, but I guess what I’m suggesting is it is just as likely that he failed to do so properly, for the same reason we think it’s plausible that professional coaches fail to properly consider statistics. Except at least the coaches have more experience and have been selected based on successful strategies, whereas Romer has not.
The paper clearly tries to take all relevant strategic considerations into account. I can’t personally vouch that he succeeds. But I haven’t heard of his result being overturned on technical grounds. That is, I haven’t heard of anyone else coming along, doing a similar analysis, and concluding that coaches actually go for it on fourth down the correct amount. For this reason, I doubt that the problem is a technical flaw in his analysis.