A Personal (Interim) COVID-19 Postmortem

I think it’s im­por­tant to clearly and pub­li­cly ad­mit when we were wrong. It’s even bet­ter to di­ag­nose why, and take steps to pre­vent do­ing so again. COVID-19 is far from over, but given my early stance on a num­ber of ques­tions re­gard­ing COVID-19, this is my at­tempt at a pub­lic per­sonal re­view to see where I was wrong.

I have been push­ing for bet­ter fore­cast­ing and prepa­ra­tion for pan­demics for years, but I wasn’t fore­cast­ing on the var­i­ous spe­cific ques­tions about Pan­demics on most plat­forms un­til at least mid-March, and I failed in sev­eral ways.

Mea Culpa

I was late to up­date about a num­ber of things, and sim­ply wrong in some cases even on the ba­sis of known in­for­ma­tion. The failures in­clude ini­tially be­ing slow to rec­og­nize the ex­tent of the threat, start­ing out dis­mis­sive about masks, be­ing more con­cerned about hos­pi­tal-based trans­mis­sion than ended up be­ing jus­tified, be­ing over­con­fi­dent in the re­sponse of the US gov­ern­ment, and in early March, over-con­fi­dently get­ting a key fact wrong about trans­mis­sion be­ing at least largely via aerosol droplet ver­sus phys­i­cal con­tact. I have a num­ber of ex­cuses, of course. Most other ex­perts agreed with my views, my grand­father passed away in Jan­uary, fol­lowed by his wife in early March, I was un­der a lot of stress, I was very busy with my per­sonal life, I was try­ing to do a num­ber of other high-pri­or­ity pro­jects, I was not pay­ing at­ten­tion to the de­tails, and so on. But pre­dic­tive ac­cu­racy doesn’t care about WHY you were wrong, es­pe­cially since there are always such ex­cuses. And the im­pact of my poor judge­ment was also likely mis­lead­ing to oth­ers in the com­mu­nity.

At the same time, I feel the per­haps ego­tis­ti­cal need to note where I was cor­rect early, and what I got right—fol­lowed by a clearer de­scrip­tion of my failures. I started say­ing there would be PPE short­ages due to COVID-19 by Jan­uary, and was writ­ing about the sup­ply chain is­sues well be­fore COVID. I sub­mit­ted this pa­per Novem­ber last year with Dave Denken­berger, which was largely finished last sum­mer, and it was ac­cepted in Fe­bru­ary, which then took 3 months to get pub­lished. The de­lay was in part due to other de­mands on my time, but in ret­ro­spect, if it had been available 3 months ear­lier, it would have been far, far more im­pact­ful.

I also un­der­stood the failure mode we ended up see­ing, and in my 2018 pa­per, dis­cussing over­con­fi­dence in claims that pan­demics would be rare, I ar­gued that among the most crit­i­cal risks was failure to re­spond to emerg­ing pan­demics which could in the­ory be con­trol­led quickly enough. On the other hand, my failure to re­al­ize that this is ex­actly what was hap­pen­ing is per­haps com­pounded by the fact that I un­der­stood the dy­nam­ics, and should have been able to iden­tify what was go­ing on.

Lastly, I main­tain I was cor­rect in warn­ing about the poorly thought out and in some cases out­right dan­ger­ous “prepa­ra­tion” in some quar­ters of the ra­tio­nal­ity com­mu­nity pro­posed in March, such as ad­vo­cat­ing use of bleach and ozone in closed ar­eas for dis­in­fec­tion. Some peo­ple in the com­mu­nity were stock­piling N-95 masks and food and buy­ing up sec­ond hand ven­tila­tors, and as I said at the time, were at best be­ing self­ish and defect­ing. On the other hand, as I men­tion be­low, I was in­suffi­ciently clear about the need for bet­ter prepa­ra­tion, and waited far too long to speak.

Some of My Mis­takes, and Re­lated Comments

Slow to rec­og­nize the ex­tent of the threat.

I said we should be very con­cerned in Jan­uary, albeit not very pub­li­cly. I took un­til early March to start sug­gest­ing that it was clear that the US would ex­pect to see large num­bers of deaths. I was skep­ti­cal of valuable efforts early on, and didn’t start re­ally pub­li­cly sound­ing the alarm and re­act­ing un­til even later. I was later than most of this com­mu­nity in rec­og­niz­ing the risks.

Skep­ti­cal about Border Closures

In a con­ver­sa­tion that started Jan 27th, I was asked about shut­ting down bor­ders to pre­vent spread. I was dis­mis­sive, in large part based on the ex­pert con­sen­sus. I’m un­sure whether this was a mis­take on the ob­ject level, since I think that at that early point, the facts were un­clear enough, and trade wars re­ally are bad. I also ex­pected re­sponse to be bet­ter, based on pre­vi­ous cases.

I do not think that bor­der shut­downs were fea­si­ble, and his­tor­i­cally they have not been. Quaran­tines at bor­ders were and are lo­gis­ti­cally im­pos­si­ble. And full bor­der clo­sures for COVID-19 were also not very effec­tive most places un­til very late in the spread, (Mon­go­lia and Viet­nam are the ex­cep­tions that dis­prove the rule.) Even late in the pan­demic spread, lots of trans­mis­sion oc­curred from places where there had been few or no cases at the time peo­ple en­tered. How­ever, when dis­cussing it, I ex­cused my early claims that it was too eco­nom­i­cally dam­ag­ing and would have been in­effec­tive by sub­sti­tut­ing a differ­ent ar­gu­ment about poli­ti­cal fea­si­bil­ity—one which I think is cor­rect, but was not my origi­nal con­sid­er­a­tion. This was bad epistemic prac­tice, and I should have been clearer that in ret­ro­spect, if they could have been put in place, travel bans would have been a much bet­ter idea. I still think my later ex­cuse, that they were poli­ti­cally im­pos­si­ble, holds up—but I had not fully thought through the ques­tion un­til well af­ter my early re­sponse.

Dis­mis­sive about masks.

The re­search on use of masks was un­clear and I don’t want to claim it was ret­ro­spec­tively ob­vi­ous, but as a mat­ter of de­ci­sion mak­ing given un­cer­tain risks, peo­ple should have started wear­ing home­made masks in pub­lic much ear­lier. We will still need to see how much im­pact pro­mot­ing mask wear­ing in pub­lic has had, but at the very least it func­tioned as a clear and im­por­tant pub­lic sig­nal that COVID was se­ri­ous, which pro­motes phys­i­cal dis­tance and other crit­i­cal fac­tors.

On the other hand, I said at the time, and still main­tain that I was cor­rect in sug­gest­ing that buy­ing up P95 and sur­gi­cal masks in Fe­bru­ary and March was defect­ing, since it was already clear that those sup­plies were needed des­per­ately in hos­pi­tals. And Fauci has now said as much (as a level-1+2 sage, in my view.). In ret­ro­spect, I think it would have been bet­ter, con­se­quen­tially, to push for cloth masks ear­lier, but cur­rent mod­el­ing and our un­der­stand­ing of spread make it clear that mask wear­ing by it­self is only marginally effec­tive. I was in­stead fo­cused on pro­mot­ing hand­wash­ing, which I think is still un­der­sold in im­por­tance, and thought that con­tinued fo­cus on masks would be a net nega­tive. I was wrong, and oth­ers here were cor­rect.

Not clear enough about the im­por­tance of prepa­ra­tion.

I’ve long said, fol­low­ing all of the ex­perts, that peo­ple should have 2 weeks sup­ply of food and ba­sic sup­plies. Espe­cially peo­ple in Cal­ifor­nia, where earth­quakes are far more com­mon than se­vere pan­demics. Fur­ther prepa­ra­tion should have been un­needed early on—but in fact, most peo­ple don’t do this, and the peo­ple who were ad­vo­cat­ing mak­ing sure that you were pre­pared for a worse out­come were cor­rect.

On the other hand, there is an ar­gu­ment I’ve seen here, and by oth­ers in the ra­tio­nal­ity com­mu­nity el­se­where, that en­courag­ing peo­ple to buy crit­i­cal sup­plies and hoard early in a crisis sends a price sig­nal to get com­pa­nies to pro­duce. The ar­gu­ment is that this type of hoard­ing masks and other PPE will con­vince man­u­fac­tur­ers to make more. I thought, and still think, that this is at least partly mi­s­un­der­stand­ing the way that price sig­nals and sup­ply chain de­lays prop­a­gate. Any­one who’s fa­mil­iar with MIT Sys­tem Dy­nam­ics’ Beer Game and the bul­lwhip effect would tell you that com­pa­nies that ramped up pro­duc­tion in re­sponse to de­mand quickly (rather than pro­jec­tions and an un­der­stand­ing of longer term de­mand) were be­ing stupid, not pru­dent, and com­pa­nies that tried this in ex­actly this area were burned in the past for do­ing so. If that isn’t clear enough, no­tice that it took a cou­ple months for the toi­let pa­per and flour “short­ages” to be worked out, de­spite the fact that there was suffi­cient sup­ply, and there were not ac­tual pro­duc­tion sup­ply short­ages. Yes, mar­kets are largely effi­cient, but they aren’t mag­i­cal ways to elimi­nate pro­duc­tion and dis­tri­bu­tion de­lays, much less to in­su­late com­pa­nies from ac­tual mar­ket dy­nam­ics—and China and other south­east Asian coun­tries had already stepped up mask pro­duc­tion mas­sively by mid-Jan­uary. Most of the cur­rent sup­ply comes from those fac­to­ries, so the sup­posed benefits of price sig­nals from buy­ing masks in Fe­bru­ary seem not to have been ac­tu­ally effec­tive in speed­ing any­thing up.

Over­sold Hospi­tal-based trans­mis­sion.

Part of my con­cern about hoard­ing of masks and other equip­ment was that I thought we would once again see a pat­tern of large trans­mis­sion events be­ing cen­tered around hos­pi­tals. Thank­fully, this didn’t hap­pen—hos­pi­tals have got­ten far bet­ter at iso­la­tion of pa­tients, and they shut down non-es­sen­tial ser­vices early. We did still see many, many cases and deaths in hos­pi­tal staff, and this was very clearly in large part due to a lack of sup­ply of PPE. Still, it wasn’t the crit­i­cal lo­cus of spread I ex­pected it to be.

Over­con­fi­dence in the re­sponse of (cer­tain agen­cies in) the US gov­ern­ment.

This was a huge mis­take on my part. I have been con­cerned about the cur­rent ad­minis­tra­tion for years, have re­peat­edly warned that it is de­stroy­ing gov­ern­ment agen­cies. De­spite that, I was (in ret­ro­spect very un­rea­son­ably) still con­fi­dent that the CDC was go­ing to han­dle the situ­a­tion well. They had hand­books on in­fluenza pan­demic pre­pared­ness, I had per­son­ally dis­cussed pan­demic pre­pared­ness plans with se­nior peo­ple at CDC just a few years ago, and I was over­con­fi­dent in the abil­ity to re­spond. Based on that, in turn, I was con­fi­dent that the level of con­cern be­ing voiced by the CDC was a re­flec­tion of their plan­ning and on­go­ing prepa­ra­tion. The CDC has planned for prepa­ra­tion for this ex­act case for years, and I as­sumed they would carry out those plans. I was wrong.

It seems, though it is still some­what un­clear, that cen­ter di­rec­tors were told by the di­rec­tor and the head of HHS that they needed not to speak out about the risks, spe­cific recom­men­da­tions were ve­toed, and (eas­ily the worst screw up,) they let the FDA ban pri­vate tests, seem­ingly at the di­rec­tion of the ad­minis­tra­tion, to hide the ex­tent of the spread. I’m still con­fused by the level of non-re­ac­tion among non-poli­ti­cal SES staff and GS-14s. We have seen many peo­ple in var­i­ous agen­cies come for­ward with com­plaints dur­ing this ad­minis­tra­tion, but CDC seems to have just dropped the ball on their re­sponse. We will likely see in the com­ing years how much this was due to cen­tral di­rec­tives not to re­act, ver­sus alack of cen­tral di­rec­tives to re­act, there­fore failing due to pas­sivity. I still want to as­sume the former, but that’s in large part self-jus­tifi­ca­tion of my prior views.

I was wrong in try­ing to defend the CDC’s over­all re­sponse in March. It definitely isn’t as clear as I thought at the time that they were, and would be, net pos­i­tive. I do think that the emer­gence of Fauci as al­most a na­tional hero has been very helpful in get­ting peo­ple to listen to ex­pert recom­men­da­tions, even if this did come very late. This is a point on the side of get­ting most peo­ple to listen more and at­tack less. On the other hand, Less­wrong was over­all bet­ter pre­pared be­cause of their skep­ti­cism, so at the very least I was talk­ing to the wrong crowd to defend them, and more likely should have been quicker to judge their ac­tions as dan­ger­ous my­self.

The FDA also sur­prised me with how badly they did, albeit the sur­prise was less se­vere be­cause I had lower ex­pec­ta­tions. I thought they were get­ting less dan­ger­ous to US pub­lic health given the pre­vi­ous pushes to re­duce reg­u­la­tion by the cur­rent ad­minis­tra­tion. Scott Got­tlieb was there for two years, and was prob­a­bly the only Trump nom­i­nee I was ac­tu­ally su­per-happy about. Un­for­tu­nately, he left (a fact I wasn’t pay­ing at­ten­tion to,) and it turns out that the in­com­pe­tence of a se­quence of new di­rec­tors and rapid changes left the FDA even less pre­pared that they would have been. I would have ex­pected a doc­tri­naire Repub­li­can ap­poin­tee to seize the op­por­tu­nity of a crisis to re­duce reg­u­la­tion, and in­stead it seems they did noth­ing but block crit­i­cal test­ing work for months.

I’ve long con­sid­ered my­self skep­ti­cal of gov­ern­ment agen­cies abil­ities, and lean fairly heav­ily liber­tar­ian in many ways—albeit less than most oth­ers at less­wrong. I was still sur­prised by the level of on­go­ing, per­haps even mal­i­cious in­com­pe­tence of the cur­rent ad­minis­tra­tion. I’m still un­clear if this is a Han­lon-dodge, or if they re­ally have bro­ken the US gov­ern­ment so badly, so quickly. Other gov­ern­ments man­aged this far less poorly, so I’m un­clear how gen­er­al­iz­able the les­son is that gov­ern­ments are bad at ev­ery­thing. But I am glad I left the US.

Be­ing a jerk com­ment­ing on a post at­tack­ing the CDC

Given that I’m post­ing a ret­ro­spec­tive, there is a differ­ent type of mis­take I made that I also need to ad­dress. In a less­wrong thread sev­eral months ago, there were a num­ber of claims made about the CDC’s re­sponse. I re­sponded that I thought the post was an in­fo­haz­ard, would very plau­si­bly lead to many more peo­ple dy­ing, and as such, the posters should have asked for feed­back from some­one who could vet con­cerns about this, and that it should be taken down by site ad­minis­tra­tors. This was stupid, and I have apol­o­gized there, along with lay­ing out what I hope is a fair anal­y­sis of what I know I did wrong, and what I still think I was cor­rect about.

Spec­u­la­tion about Causes

There are lots of things I did wrong.

First, I think I was too close to the situ­a­tion. I had spent a ton of time look­ing at the US’s sys­tem speci­fi­cally, and writ­ing about the closely re­lated -topic of in­fluenza pan­demics in my dis­ser­ta­tion, then do­ing work for Open Philan­thropy on GCBRs. All of this was dur­ing the Obama ad­minis­tra­tion. I left the US a bit af­ter Trump was elected, partly for that rea­son, and worked on re­lated top­ics that had less to do with US policy. I’d like to say that’s why I didn’t up­date, but to be hon­est, I think I was just be­ing stupid in ac­cept­ing my cached thoughts about the risk and best re­sponses, in­stead of re-eval­u­at­ing.

I also had too-strong pri­ors and “ex­pert” ideas to be prop­erly fox-like in my pre­dic­tions, and not quick enough to up­date about how things were ac­tu­ally go­ing based on the data. Be­cause I was slow to move from the base-rate, I un­der­es­ti­mated the sever­ity of COVID-19 for too long. I’m un­sure how to fix that, since most of the time it’s the right move, and pay­ing at­ten­tion to ev­ery new event is very ex­pen­sive in terms of men­tal en­ergy. (Sugges­tions wel­come!)

I also gave too much weight to oth­ers’ fore­casts. Good Judge­ment’s pre­dic­tions were WAY op­ti­mistic about this early on, and I was not fore­cast­ing the ques­tion, but I was as­sum­ing that their ag­gre­gate guess was bet­ter than that of in­di­vi­d­u­als, es­pe­cially peo­ple who aren’t fore­cast­ers. This is usu­ally cor­rect, but here it was a mis­take. (I now think that su­perfore­cast­ing is ma­te­ri­ally worse than I hoped it would be at notic­ing rare events early.) I also fol­lowed the herd too much from ex­pert cir­cles, and my twit­ter feed from in­fec­tious dis­ease epi­demiol­ogy cir­cles was be­hind even my slow self in rec­og­niz­ing that this was a in­cip­i­ent dis­aster back in March.

Conclusion

COVID-19 went badly in some places, and went dis­as­trously in oth­ers. This was largely pre­dictable, and I failed to no­tice early enough. (The US is in deep, deep trou­ble, and this will con­tinue for quite a while longer, with myr­iad longer term effects on the global econ­omy, and on global sta­bil­ity of other types.) I’m chas­tened about the poorly cal­ibrated over­con­fi­dence of my ex­pert opinion.

I’m also partly un­sure what the best next steps are for bet­ter-cal­ibra­tion. One key thing I did, sev­eral years ago, was ex­plic­itly try to rely more on other peo­ple’s views in the ra­tio­nal­ity com­mu­nity to guide my de­ci­sions, and provide a clear source of feed­back. I didn’t do this as much as I should have in this case. (On the other hand, it was a large part of why I rec­og­nized the mis­take as quickly as I did, albeit later than I could have—so it was at least a par­tial suc­cess.)

I’m hop­ing that this ex­er­cise is an­other way in which think­ing through the situ­a­tion gives me a valuable chance to re­flect, and that I can get fur­ther feed­back. I also hope that it’s use­ful for oth­ers to per­haps learn from, but I’m un­sure how trans­fer­able the les­sons of my failures are.