As mentioned in the post, I think it’s personally helpful to look back, and is a critical service to the community as well. Looking back at looking back, there are things I should add to this list—and even something (hospital transmission) which I edited more recently because I have updated against having been wrong about in this post—but it was, of course, an interim postmortem, so both of these types of post-hoc updates seem inevitable.
I think that the most critical lesson I learned was to be more skeptical of information sources generally—even the most accurate, including superforecasters and the rationalist community, are fallible in ways which are somewhat predictable, and hard to evaluate prior to knowing the ground truth. This both highlights the value of staying uncertain and entertaining multiple hypotheses, and the importance of keeping diverse information sources available. The points made by John Wentworth in his comment about the need to do expensive updates was also very clear and valuable.
I certainly think additional posts of this type, by myself and by others, would provide value—and I could see it being its own genre. Unfortunately, there have been very few. I am happy to see several projects looking back at the community’s reactions, successes, and failures, but they are still in progress. The 2020 Petrov Day postmortem and similar are also evaluating community behavior, and some have evaluated failures in companies, but I see fairly few, and I would think we could use more, and more individual posts. (I’d hoped to write another actual after-action report, but I have been busy—an insufficient excuse—and we’re unfortunately still not post-COVID-19.)
As mentioned in the post, I think it’s personally helpful to look back, and is a critical service to the community as well. Looking back at looking back, there are things I should add to this list—and even something (hospital transmission) which I edited more recently because I have updated against having been wrong about in this post—but it was, of course, an interim postmortem, so both of these types of post-hoc updates seem inevitable.
I think that the most critical lesson I learned was to be more skeptical of information sources generally—even the most accurate, including superforecasters and the rationalist community, are fallible in ways which are somewhat predictable, and hard to evaluate prior to knowing the ground truth. This both highlights the value of staying uncertain and entertaining multiple hypotheses, and the importance of keeping diverse information sources available. The points made by John Wentworth in his comment about the need to do expensive updates was also very clear and valuable.
I certainly think additional posts of this type, by myself and by others, would provide value—and I could see it being its own genre. Unfortunately, there have been very few. I am happy to see several projects looking back at the community’s reactions, successes, and failures, but they are still in progress. The 2020 Petrov Day postmortem and similar are also evaluating community behavior, and some have evaluated failures in companies, but I see fairly few, and I would think we could use more, and more individual posts. (I’d hoped to write another actual after-action report, but I have been busy—an insufficient excuse—and we’re unfortunately still not post-COVID-19.)