Coronavirus as a test-run for X-risks

On the 27th Fe­bru­ary, I, like many of us, be­came fully aware of the dan­ger hu­man­ity was fac­ing (let’s thank ‘See­ing the Smoke’) and put my cards on the table with this:

This is partly a test run of how we’d all feel and re­act dur­ing a gen­uine ex­is­ten­tial risk. Me­tac­u­lus cur­rently has it as a 19% chance of spread­ing to billions of peo­ple, a dis­aster that would cer­tainly re­sult in many mil­lions of deaths, prob­a­bly tens of mil­lions. Not even a catas­trophic risk, of course, but this is what it feels like to be fac­ing down a 15 chance of a ma­jor global dis­aster in the next year. It is an op­por­tu­nity to un­der­stand on a gut level that, this is pos­si­ble, yes, real things ex­ist which can do this to the world. And it does hap­pen.
It’s worth think­ing that spe­cific thought now be­cause this par­tic­u­lar epistemic situ­a­tion, a 15 chance of a ma­jor catas­tro­phe in the next year, will prob­a­bly arise again over the com­ing decades. I can eas­ily imag­ine star­ing down a similar prob­a­bil­ity of dan­ger­ously fast AGI take­off, or a nu­clear war, a few months in ad­vance.

Well, now a few months have gone by and much has changed. The nat­u­ral ques­tion to ask is- what gen­eral les­sons have we learned, com­pared to that ‘par­tic­u­lar epistemic situ­a­tion’, now that we’re in a sub­stan­tially differ­ent one? What does hu­man­ity’s re­sponse to the coro­n­avirus pan­demic so far im­ply about how we might fare against gen­uine X-risks?

At a first pass, the an­swer to that ques­tion seems ob­vi­ous—not very well. The re­sponse of most usu­ally well-func­tion­ing gov­ern­ments (I’m think­ing mainly of Western Europe here) has been slow, held back by an un­will­ing­ness to com­mit all re­sources to a strat­egy and ac­cept its trade-offs, and slug­gish to re­spond to chang­ing ev­i­dence. Ad­vance prepa­ra­tion was even worse. This post gives a good sum­mary of some of those more ob­vi­ous les­sons for X-risks, fo­cussing speci­fi­cally on slow AI take­off.

As to what we ul­ti­mately blame for this slow­ness—Scott Alexan­der and Toby Ord gave as good an ac­count as any­one (be­fore the pan­demic) in blam­ing a failure to un­der­stand ex­pected value and the availa­bil­ity heuris­tic.

How­ever, many of us pre­dicted in ad­vance the dy­nam­ics that would lead coun­tries to put for­ward a slow and in­co­her­ent re­sponse to the coro­n­avirus. What I want to ex­plore now is—what has changed epistem­i­cally since I wrote that com­ment—what things have hap­pened since that have sur­prised many of us who have in­ter­nal­ised the truth of civil­i­sa­tional in­ad­e­quacy? I am look­ing for gen­er­al­ised les­sons we can take from this pan­demic, rather than spe­cific things we have learnt about the pan­demic in the last few months. I be­lieve there is one such les­son that is sur­pris­ing, which I’d like to con­vince you of.

Un­der­weight­ing Strong Reactions

My claim is that in late Fe­bru­ary/​early March, many of us did over­look or un­der­weight the pos­si­bil­ity that many coun­tries would even­tu­ally re­act strongly to coro­n­avirus—with mea­sures like lock­downs that suc­cess­fully drove R un­der 1 for ex­tended pe­ri­ods, or with in­di­vi­d­ual ac­tion that holds R near 1 in the ab­sence of any real gov­ern­ment in­ter­ven­tion. This meant we placed too much weight on coro­n­avirus go­ing un­con­tained, and were sur­prised when in many coun­tries it did not.

Whether the strong re­ac­tion is fully or only par­tially effec­tive re­mains to be seen, but the fact that this re­ac­tion oc­curred was sur­pris­ing to many of us, rel­a­tive to what we be­lieved at the start of all this—I know that it sur­prised me.

I will first pre­sent the ex­am­ples of pre­dic­tions, some from peo­ple on LessWrong or ad­ja­cent groups and some from gov­ern­ment sci­en­tists, which all ei­ther fore­told worse out­comes by now, more fee­ble re­sults from in­ter­ven­tions, lower com­pli­ance with in­ter­ven­tions, that in­ter­ven­tions wouldn’t even be im­ple­mented or pre­dicted bad out­comes that are not yet ruled out but now look much less likely than they did.

I will then put for­ward an ex­pla­na­tion for these mis­takes—some­thing I named (in April) the ‘Mori­t­uri Nolu­mus Mori’ (‘We who are about to die don’t want to’) effect, in refer­ence to the Disc­world novel The Last Hero: that most gov­ern­ments and in­di­vi­d­u­als have a con­sis­tent, short-term aver­sion to dan­ger which is stronger than many of us sus­pected, though not sus­tain­able in the ab­sence of an im­mi­nent threat. I’ll first go through the in­cor­rect pre­dic­tions and then give my favoured ex­pla­na­tion. If I am cor­rect that many of us (and also many sci­en­tists and poli­cy­mak­ers) missed the im­por­tance of the MNM effect, it should in­crease our con­fi­dence that, in situ­a­tions where there is some warn­ing, there are fairly ba­sic fea­tures of our psy­chol­ogy and in­sti­tu­tions that do get in the way of the very worst out­comes. How­ever, the MNM effect is limited and will not help in any situ­a­tion where ad­vance plan­ning or re­spond­ing to things above and be­yond im­me­di­ate in­cen­tives are re­quired.

I con­sider the MNM effect to be mostly com­pat­i­ble with Zvi’s ‘Govern­ments Most Places Are Ly­ing Liars With No Abil­ity To Plan or Phys­i­cally Rea­son.’ (I do think that claim is too Amer­ica-cen­tric, and ‘no abil­ity to plan/​rea­son’ is hy­per­bole if ap­plied to Europe or even the UK, let alone e.g. Taiwan). The MNM effect is what we credit in­stead of clever plan­ning or rea­son­ing, for why things aren’t as bad as they could be—the differ­ences be­tween e.g. Amer­ica and Ger­many are due to any level of plan­ning at all, not bet­ter plan­ning.

Notic­ing Confusion

Things (es­pe­cially in the US) are suffi­ciently bad right now that it is difficult to re­mem­ber that many of us put sig­nifi­cant weight on things already be­ing worse than they cur­rently are—but as I will show that was the case.

Some peo­ple’s ini­tial pre­dic­tions were that R would not be driven sub­stan­tially be­low 1 for any ex­tended pe­riod, any­where, ex­cept with a Wuhan style lock­down. Robin Han­son seem­ingly claimed this on March 19: ‘So even if you see China policy as a suc­cess, you shouldn’t have high hopes if your gov­ern­ment merely copies a few sur­face fea­tures of China policy.‘ In that ar­ti­cle Han­son was clearly refer­ring to ‘most gov­ern­ments’ that aren’t China as be­ing un­likely to sup­press with­out adopt­ing a deep mimicry of China’s policy—in­clud­ing weld­ing peo­ple into their flats and forcible case iso­la­tion. Yet, two months later there are many coun­tries, from New Zealand to Ger­many, which have sim­ply copied some but not all fea­tures of the Chi­nese policy while achiev­ing ini­tial sup­pres­sion.

More re­cently, Han­son up­dated to speak­ing more speci­fi­cally about the USA: (in re­sponse to a graphic show­ing sev­eral ex­am­ples of suc­cess­ful sup­pres­sion in Europe and Asia) ‘Yes, you know that other na­tions have at times won wars. Even so, you must de­cide if to choose peace or war.’ Go­ing from ‘most west­ern coun­tries’ to ‘Amer­ica’ counts as an op­ti­mistic up­date.

But miti­ga­tion mea­sures (which Han­son calls ‘peace’) have also worked out less dis­as­trously than our worst fears sug­gested be­cause of stronger-than-ex­pected in­di­vi­d­ual ac­tion. See e.g. this ar­ti­cle about Swe­den:

Ul­ti­mately, Swe­den shows that some of the worst fears about un­con­trol­led spread may have been overblown, be­cause peo­ple will act them­selves to stop it. But, equally, it shows that crit­i­cisms of lock­downs tend to ig­nore that the real coun­ter­fac­tual would not be busi­ness as usual, nor a rapid at­tain­ment of herd im­mu­nity, but a slow, bru­tal, and un­con­trol­led spread of the dis­ease through­out the pop­u­la­tion, kil­ling many peo­ple. Judg­ing from serolog­i­cal data and deaths so far, it is the speed of deaths that peo­ple who warned in favour of lock­downs got wrong, not the scale.

This re­mark about Swe­den is ap­pli­ca­ble more gen­er­ally—the worst case sce­nario for al­most ev­ery coun­try seems to be R around 1.5 at this point—see this map from Epi­demic Fore­cast­ing. True ex­plo­sive spread is very rare across the world, but was be­ing dis­cussed as a real pos­si­bil­ity in early March even in Europe. Again, the re­sponse is not good enough to out­right re­verse the un­fold­ing dis­aster, but it is still strong enough to ar­rest ex­plo­sive spread.

Fo­cussing on the UK, which had a badly de­layed re­sponse and a highly im­perfect lock­down, we can see that even there R was driven sub­stan­tially be­low 1 and hos­pi­tal ad­mis­sions with Covid-19 (which are the most re­li­able short-term proxy for in­fec­tion through­out the over­all pan­demic) are at 13% of their peak. Lon­don did not ex­ceed its ICU ca­pac­ity de­spite pre­dic­tions that it would from gov­ern­ment mod­el­lers.

Another way of get­ting at this dis­joint is to just look at the num­bers and see if we still ex­pect the same num­ber of peo­ple to die. Wei Dai ini­tially (1st March) pre­dicted 190-760 mil­lion peo­ple would even­tu­ally die from coro­n­avirus with 50% of the world in­fected. The more re­cent top-rated com­ment by Orthonor­mal points out that cur­rent ev­i­dence points against that. Good Judg­ment rates the prob­a­bil­ity that more than 80 mil­lion will die as 1%. A re­cent pa­per by Im­pe­rial Col­lege sug­gested that the Europe-wide lock­downs have so far saved 3 mil­lion lives with­out ac­count­ing for the fact that deaths in an un­miti­gated sce­nario would have been higher due to a lack of in­ten­sive care beds. Re­gard­less of what hap­pens next, would we have pre­dicted that in early March?

Th­ese mis­takes have not been limited to the LessWrong com­mu­nity—one of the rea­sons for the afore­men­tioned de­lay be­fore the UK called the lock­down was that UK be­havi­oural sci­en­tists ad­vis­ing the gov­ern­ment were near cer­tain that stringent lock­down mea­sures would not be obeyed to the nec­es­sary de­gree and lock­downs in the rest of Europe were in­stead im­ple­mented ‘more for soli­dar­ity rea­sons’. In the end it turned out that com­pli­ance was in­stead ‘higher than ex­pected’. The at­ti­tude in most of Europe in early March was that full lock­downs were com­pletely in­fea­si­ble. Then they were im­ple­mented.

Another way of get­ting at this ob­ser­va­tion is to note the peo­ple who have pub­li­cly recorded their sur­prise or shift in be­lief as these events have un­folded. I have writ­ten sev­eral com­ments with ear­lier ver­sions of this claim, start­ing two months ago. Wei Dai no­tably up­dated in the di­rec­tion of think­ing coro­n­avirus would reach a smaller frac­tion of the pop­u­la­tion, af­ter read­ing this pre­scient blog­post:

The in­ter­ven­tions of en­forced so­cial dis­tanc­ing and con­tract trac­ing are ex­pen­sive and in­evitably en­tail a cur­tail­ment of per­sonal free­dom. How­ever, they are achiev­able by any suffi­ciently mo­ti­vated pop­u­la­tion. An in­crease in trans­mis­sion *will* even­tu­ally lead to con­tain­ment mea­sures be­ing ramped up, be­cause ev­ery mod­ern pop­u­la­tion will take dra­co­nian mea­sures rather than al­low­ing a health care melt­down. In this sense COVID-19 in­fec­tions are not and will prob­a­bly never be a full-fledged pan­demic, with un­re­stricted in­fec­tion through­out the world. It is un­likely to be al­lowed to ever get to high num­bers again in China for ex­am­ple. It will always in­stead be a se­ries of lo­cal epi­demics.

In a re­cent pod­cast, Rob Wiblin and Tara Kirk Sell were dis­cussing what they had re­cently changed their minds about. They picked out the same thing:

Robert Wiblin: Has the re­sponse af­fected your views on what poli­cies are nec­es­sary or should be pri­ori­tized for next time?
Tara Kirk Sell: The fact that “Stay-at-home or­ders” are ac­tu­ally pos­si­ble in the US and seem to work… I had not re­ally had a lot of faith in that be­fore and I feel like I’ve been sur­prised. But I don’t want “Stay-at-home or­ders” to be the way we deal with pan­demics in the fu­ture. Like great, it worked, but I don’t want to do this again.

Or this from Zvi:

5. Fewer than 3 mil­lion US coro­n­avirus deaths: 90%
I held. Again, we saw very good news early, so to get to 3 mil­lion now we’d need full sys­tem col­lapse to hap­pen quickly. It’s definitely still pos­si­ble, but I’m guess­ing we’re now more like 95% to avoid this than 90%.

Lastly, we have the news from the cur­rent hard­est-hit places, like Man­hat­tan, which have already hit par­tial herd im­mu­nity and show ev­ery sign of be­ing able to con­tain coro­n­avirus go­ing for­ward even with im­perfect mea­sures.

The Mori­t­uri Nolu­mus Mori effect

Many of these facts (in par­tic­u­lar the rea­son that 100 mil­lion plus dead is effec­tively ruled out) have mul­ti­ple ex­pla­na­tions. For one, the ear­liest data on coro­n­avirus im­plied the hos­pi­tal­iza­tion rate was 10-20% for all age groups, and we now know it is sub­stan­tially lower (that tweet by an au­thor of the Im­pe­rial Col­lege pa­per, which es­ti­mated a hos­pi­tal­iza­tion rate of 4.4%). This means that if hos­pi­tals were en­tirely un­able to cope with the num­ber of pa­tients, the IFR would be in the range of 2%, not 20% ini­tially im­plied.

How­ever, the rest of our in­for­ma­tion about the char­ac­ter­is­tics of the virus in early March- the es­ti­mate of R0 and ‘stan­dard’ IFR, were fairly close to the mark. Our pre­dic­tions were work­ing off of rea­son­able data about the virus. Any pre­dic­tion made then about the num­ber of peo­ple who would be in­fected isn’t af­fected by this hos­pi­tal­iza­tion rates con­founder, nor is any pre­dic­tion about what mea­sures would be im­ple­mented. So there must be some other rea­son for these mis­takes—and a com­mon thread among nearly all the in­ac­cu­rate pes­simistic pre­dic­tions was that they un­der­es­ti­mated the force­ful­ness, though not the level of forethought or plan­ning, be­hind miti­ga­tion or sup­pres­sion mea­sures. As it is writ­ten,

“Brains don’t work that way. They don’t sud­denly su­per­charge when the stakes go up—or when they do, it’s within hard limits. I couldn’t calcu­late the thou­sandth digit of pi if some­one’s life de­pended on it.”

The Mori­t­uri Nolu­mus Mori effect, as a re­minder, is the the­sis that gov­ern­ments and in­di­vi­d­u­als have a con­sis­tent, short-term re­ac­tion to dan­ger which is stronger than many of us sus­pected, though not sus­tain­able in the ab­sence of an im­mi­nent threat. This effect is just such a hard limit—it can’t do very much ex­cept work as a stronger than ex­pected brake. And some­thing like it has been pro­posed as an ex­pla­na­tion, not just by me two months ago but by Will MacAskill and Toby Ord, for why we have already avoided the worst dis­asters. Here’s Toby’s re­cent in­ter­view:

Learn­ing the right les­sons will in­volve not just iden­ti­fy­ing and patch­ing our vuln­er­a­bil­ities, but point­ing to­wards strengths we didn’t know we had. The un­prece­dented mea­sures gov­ern­ments have taken in re­sponse to the pan­demic, and the pub­lic sup­port for do­ing so, should make us more con­fi­dent that when the stakes are high we can take de­ci­sive ac­tion to pro­tect our­selves and our most vuln­er­a­ble. And when faced with truly global prob­lems, we are able to come to­gether as in­di­vi­d­u­als and na­tions, in ways we might not have thought pos­si­ble. This isn’t about be­ing self-con­grat­u­la­tory, or ig­nor­ing our mis­takes, but in see­ing the glim­mers of hope in this hard­ship.

Will MacAskill made refer­ence to the MNM effect in a pre-coro­n­avirus in­ter­view, ex­plain­ing why he puts the prob­a­bil­ity of X-risks rel­a­tively low.

Se­cond then, is just think­ing in terms of the ra­tio­nal choice of the main ac­tors. So what’s the will­ing­ness to pay from the per­spec­tive of the United States to re­duce a sin­gle per­centage point of hu­man ex­tinc­tion whereby that just means the United States has three hun­dred mil­lion peo­ple. How much do they want to not die? So as­sume the United States don’t care about the fu­ture. They don’t care about peo­ple in other coun­tries at all. Well, it’s still many trillions of dol­lars is the will­ing­ness to pay just to re­duce one per­centage point of ex­is­ten­tial risk. And so you’ve got to think that some­thing’s gone wildly wrong, where peo­ple are mak­ing such in­cred­ibly ir­ra­tional de­ci­sions.

Bill Gates also referred to this effect.

I also think that the MNM effect is the main rea­son why both Me­tac­u­lus and su­perfore­cast­ers con­sis­tently pre­dicted deaths will stay be­low 10 mil­lion, im­ply­ing a very slow burn, nei­ther sup­pres­sion nor full herd im­mu­nity, right across most of the world.

The Con­trol System

From Slat­estar­codex:

Is there a pos­si­bil­ity where R0 is ex­actly 1? Seems un­likely – one is a pretty spe­cific num­ber. On the other hand, it’s been weirdly close to one in the US, and wor­ld­wide, for the past month or two. You could imag­ine an un­for­tu­nate con­trol sys­tem, where ev­ery time the case count goes down, peo­ple stop wor­ry­ing and go out and have fun, and ev­ery time the case count goes up, peo­ple freak out and stay in­doors, and over­all the new case count always hov­ers at the same rate. I’ve never heard of this hap­pen­ing, but this is a novel situ­a­tion.

One more spec­u­la­tive con­se­quence of the MNM effect is that a re­ac­tive, strong push against un­con­trol­led pan­demic spread is a good ex­pla­na­tion for why Rt tends to ap­proach 1 in coun­tries with­out a co­or­di­nated gov­ern­ment re­sponse, like the United States, and the more co­or­di­nated the re­sponse the fur­ther be­low 1 Rt can be pushed. A pri­ori, we might ex­pect that there is some ‘min­i­mal de­fault level’ of re­sponse that leads to Rt be­ing de­creased from R0, 3-4, to some much lower value—but why is the barom­e­ter set around 1? It’s not a co­in­ci­dence, as Zvi points out.

When­ever some­thing lands al­most ex­actly on the only in­flec­tion point, in this case R0 of one where the rate of cases nei­ther in­creases nor de­creases, the right re­ac­tion is sus­pi­cion.
In this case, the ex­pla­na­tion is that a con­trol sys­tem is in play. Peo­ple are pay­ing tons of at­ten­tion to when things are ‘get­ting bet­ter’ or ‘get­ting worse’ and ad­just­ing be­havi­our, both legally re­quired ac­tions and vol­un­tary ac­tions.

The MNM effect is ap­par­ently so pre­dictable that, with short-ish term feed­back, it can form a con­trol sys­tem. The other end of this con­trol sys­tem is all the usual cog­ni­tive and in­sti­tu­tional bi­ases that pre­vent us from tak­ing these events se­ri­ously and ac­tu­ally plan­ning for them.

It is pos­si­ble this is the first time such a con­trol sys­tem has formed to miti­gate a wide­spread dis­aster. Disasters of this size are rare through­out his­tory. Add to this the fact that such con­trol sys­tems can only form when the threat un­folds and changes over sev­eral months, giv­ing peo­ple time to veer be­tween in­cau­tion and cau­tion. Mean­while, the short term feed­back which gov­ern­ments and peo­ple can ac­cess about the progress of the epi­demic is rel­a­tively new—bet­ter data col­lec­tion and mass me­dia make mod­ern pop­u­la­tions much more sen­si­tive to the cur­rent level of threat than those through­out his­tory. Re­mem­ber­ing that noone knows ex­actly where or when the Span­ish Flu be­gan high­lights that good real-time mon­i­tor­ing of a pan­demic is an ex­tremely new thing.

In our cur­rent situ­a­tion of equil­ibrium cre­ated by a con­trol sys­tem, the re­main­ing un­cer­tain­ties are: can we do bet­ter than the equil­ibrium po­si­tion? (so­ciolog­i­cal and poli­ti­cal) and how bad is the equil­ibrium po­si­tion? (mainly a mat­ter of the dis­ease dy­nam­ics). It seems to me, the equil­ibrium prob­a­bly ends in par­tial herd im­mu­nity (nowhere near 75% ‘full herd im­mu­nity’, be­cause of MNM). This in­volves health­care sys­tems strug­gling to cope to some ex­tent along the way. The US is es­sen­tially bound for equil­ibrium—but what that en­tails is not clear. I could imag­ine the equil­ibrium hold­ing Rt near 1 even in the ab­sence of any gov­ern­ment fore­sight or plan­ning but it doesn’t seem very likely, as some com­menters pointed out. More likely it ends with par­tial herd im­mu­nity.

How­ever, there is still a push away from this equil­ibrium in Europe (e.g. at­tempts to use na­tional-level trac­ing and test­ing pro­grams). This push is not that strong and de­pends on in­di­vi­d­u­als stick­ing to so­cial dis­tanc­ing rules. Euro­pean lock­downs brought Rt down to be­tween 0.6 and 0.8, no­tice­ably be­low 1, in­di­cat­ing that they beat the equil­ibrium to some de­gree for a while. Rt got down to 0.4 in Wuhan, sug­gest­ing great suc­cess in beat­ing the equil­ibrium.

That is the other les­son—any level of gov­ern­ment fore­sight or plan­ning adds on to the already ex­ist­ing MNM effect—wit­ness how foot traf­fic lev­els dra­mat­i­cally de­clined be­fore lock­downs were in­sti­tuted, or even if they were never in­sti­tuted, right across the world. The effects are ad­di­tive. So if the de­fault holds Rt near 1, then a few ex­tra ac­tions by a gov­ern­ment able to look some de­gree into the fu­ture can make all the differ­ence.


I con­sider that the num­ber of pre­dic­tions that have already been falsified or ren­dered un­likely is suffi­cient to es­tab­lish that the MNM effect ex­ists, or is stronger than many of us thought early on (I don’t imag­ine there were many peo­ple who would have de­nied the MNM effect ex­ists at all, i.e. ex­pected us to just walk will­ingly to our deaths). ‘Dumb re­open­ing’ as is hap­pen­ing the US, as a suc­ces­sor to lock­downs that have pushed R to al­most ex­actly 1, is con­sis­tent with what I have claimed—that our re­li­able and pre­dictable short-term re­ac­tivity (gov­ern­men­tal and in­di­vi­d­ual) and de­sire to not die, the Mori­t­uri Nolu­mus Mori effect, serves as a brake against the very worst out­comes. What next?

Con­ceiv­ably, the con­trol sys­tem could keep run­ning, and R could stay near 1 per­pet­u­ally even with no effec­tive plan­ning or well-en­forced lock­downs, or there could be a slow grind as the virus spreads up to a par­tial herd im­mu­nity thresh­old—ei­ther way, the MNM effect is there, screen­ing off some out­comes that looked likely in early March, such as a sin­gle sharp peak. Similarly, the MNM effect gives a helping hand to at­tempts at real strat­egy. Some gov­ern­ments that are com­pe­tent in the face of mas­sive threats but slow to re­act (such as Ger­many) did bet­ter than ex­pected be­cause of the cau­tion of cit­i­zens who started re­strict­ing their move­ments be­fore lock­down and who now aren’t tak­ing full ad­van­tage of re­opened pub­lic spaces.

From the per­spec­tive of pre­dict­ing fu­ture X-risks, the over­all out­come of this pan­demic is less in­ter­est­ing than the fact that there has been a con­sis­tent, unan­ti­ci­pated push from re­ac­tive ac­tions against the spread of the virus. Then there is a fur­ther, also rele­vant is­sue of whether coun­tries can beat the equil­ibrium (of R be­ing held at near 1 or just above 1) and do bet­ter than the MNM effect man­dates. So far, Europe spent a while beat­ing equil­ibrium (with R dur­ing lock­down at 0.6-0.8) and China drove R down even fur­ther.

The first re­main­ing un­cer­tainty is: can a spe­cific coun­try/​the world as a whole do bet­ter than this equil­ibrium po­si­tion? We do have some per­ti­nent ev­i­dence to an­swer this in the form of the su­perfore­caster pre­dic­tions and, though it is con­founded by the next un­cer­tainty, from dis­ease mod­el­ling. The in­sights of dis­ease mod­el­ling should shed light on the ques­tion: how bad is this equil­ibrium po­si­tion? If we knew this we would have a bet­ter sense of what the rea­son­able worst case sce­nario is for coro­n­avirus, but that is not im­por­tant from an x-risk per­spec­tive.

This makes it clear what kinds of ev­i­dence are worth look­ing out for. We should look at the perfor­mance of ar­eas of the world where there is lit­tle ad­vance plan­ning, but nev­er­the­less the peo­ple are in­formed about the level of day-to-day dan­ger and lead­ers don’t ac­tively op­pose in­di­vi­d­ual efforts at safety. Parts of the United States fit the bill. See­ing the even­tual out­comes in these ar­eas, when com­pared to some ini­tial pre­dic­tions about just how bad things could get, will give us an idea of the ex­tra help pro­vided by the MNM effect. Then, with that as our baseline, we can see how many coun­tries do bet­ter to judge the fur­ther help pro­vided by plan­ning or an ac­tual strat­egy.

Im­pli­ca­tions for X-risks

The most ba­sic les­son that should be learned from this dis­aster is, of course, that for the mo­ment we are in­ad­e­quate—un­able to co­or­di­nate as long as there is any un­cer­tainty about what to do, and un­able to mean­ingfully plan in ad­vance for plau­si­ble near-term threats like those from pan­demics. We should of course re­mem­ber that not enough fo­cus is put on long-term risks, that our in­sti­tu­tions are flawed in deal­ing with them.

Covid-19 shows that there can still be a strong re­ac­tion once it is clear there is dis­aster com­ing. We have some idea already just how strong this re­ac­tion is. We have less idea how effec­tive it will end up be­ing. In Fe­bru­ary and March, we of­ten ob­served a kind of plu­ral­is­tic ig­no­rance, where even ex­perts rais­ing the alarm did so in a way that was muted and seem­ingly aimed at ‘not caus­ing panic’.

Robert Wiblin: I think part of what was go­ing on was per­haps peo­ple wanted to pro­mote this idea of “Don’t panic” be­cause they were wor­ried that the pub­lic would panic and they felt that the way to do that was re­ally to talk down the risk a lot and then it kind of got a bit out of con­trol, but I’m not sure how big the risk of… It seems like what’s ended up hap­pen­ing is much worse than the pub­lic pan­ick­ing in Jan­uary. Or maybe I just haven’t seen what hap­pens when the pub­lic re­ally pan­ics. I guess peo­ple pan­icked later and it wasn’t that bad.

Sup­pose this dy­namic ap­plies in a fu­ture dis­aster. We might ex­pect to see a sud­den phase change from in­differ­ence to panic de­spite the fact that trou­ble was already loom­ing any­way and no new in­for­ma­tion has ap­peared.

If there is enough fore­warn­ing be­fore the dis­aster oc­curs that a phase shift in at­ti­tudes can take place, we will re­act hard. Sup­pose the R0 of Coron­avirus had been 1.5-2, and the rest of our re­sponse had been oth­er­wise the same—sup­pres­sion mea­sures taken in the US and el­se­where would have worked perfectly even though we were sleep­walk­ing to­wards dis­aster as re­cently as three weeks be­fore. The only rea­son this didn’t hap­pen is be­cause of con­tin­gent facts about this par­tic­u­lar virus. On the other hand, there are mag­ni­tudes of dis­aster which the MNM effect is clearly in­ad­e­quate for—sup­pose the R0 had been 8.

Per­haps the MNM effect is stronger for a dis­aster, like a pan­demic, for which there is some de­gree of his­tor­i­cal mem­ory and evolved emo­tions and in­tu­itions around things like pu­rity and dis­gust which can take over and in­fluence our risk-miti­ga­tion be­havi­our. Maybe tech­nolog­i­cal dis­asters that don’t have the same deep evolu­tion­ary routes, like nu­clear war, or X-risks like un­al­igned AGI that have liter­ally never hap­pened be­fore, would not evoke the same strong, con­sis­tent re­ac­tion be­cause the threat is even less com­pre­hen­si­ble.

Nev­er­the­less, one could imag­ine a slow AI take­off sce­nario with a lot of the same char­ac­ter­is­tics as coro­n­avirus, where the MNM effect steps in at the last mo­ment:

It takes place over a cou­ple of years. Every day there are slight in­creases in some rele­vant warn­ing sign. A group of safety peo­ple raise the alarm but are mostly ig­nored. There are smaller scale dis­asters in the run-up, but peo­ple don’t learn their les­son (analo­gous to SARS-1 and MERS). Ma­jor news orgs and gov­ern­ment an­nounce there is noth­ing to worry about (analo­gous to ini­tial state­ments about masks and travel bans). Then there is a sud­den change in at­ti­tudes for no ob­vi­ous rea­son. At some point ev­ery­one freaks out—bans and re­stric­tions on AI de­vel­op­ment, right be­fore the crisis hits. Or, pos­si­bly, right when it is already too late.

The les­son to be learned is that there may be a phase shift in the level of dan­ger posed by cer­tain X-risks—if the amount of ad­vance warn­ing or the speed of the un­fold­ing dis­aster is above some min­i­mal thresh­old, even if that thresh­old would seem like far too lit­tle time to do any­thing given our pre­vi­ous in­ad­e­quacy, then there is still a chance for the MNM effect to take over and avert the worst out­come. In other words, AI take­off with a small amount of fore­warn­ing might go a lot bet­ter than a sce­nario where there is no fore­warn­ing, even if past perfor­mance sug­gests we would do noth­ing use­ful with that fore­warn­ing.

More spec­u­la­tively, I think we can see the MNM effect’s in­fluence in other set­tings where we have con­sis­tently avoided the very worst out­comes de­spite sys­tem­atic in­ad­e­quacy—An­ders Sand­berg refer­enced some­thing like it when he was dis­cussing the prob­a­bil­ity of nu­clear war. There have been many near misses when nu­clear war could have started, im­ply­ing that we can’t have been lucky over and over. In­stead that there has been a stronger skew to­wards in­ter­ven­tions that halt dis­aster at the last mo­ment, com­pared to not-the-last-mo­ment:

Robert Wiblin: So just to be clear, you’re say­ing there’s a lot of near misses, but that hasn’t up­dated you very much in fa­vor of think­ing that the risk is very high. That’s the re­verse of what I ex­pected.
An­ders Sand­berg: Yeah.
Robert Wiblin: Ex­plain the rea­son­ing there.
An­ders Sand­berg: So imag­ine a world that has a lot of nu­clear war­heads. So if there is a nu­clear war, it’s guaran­teed to wipe out hu­man­ity, and then you com­pare that to a world where is a few war­heads. So if there’s a nu­clear war, the risk is rel­a­tively small. Now in the first dan­ger­ous world, you would have a very strong deflec­tion. Even get­ting close to the state of nu­clear war would be strongly dis­fa­vored be­cause most his­to­ries close to nu­clear war end up with no ob­servers left at all.
In the sec­ond one, you get the much weaker effect, and now over time you can plot when the near misses hap­pen and the num­ber of nu­clear war­heads, and you ac­tu­ally see that they don’t be­have as strongly as you would think. If there was a very strong an­thropic effect you would ex­pect very few near misses dur­ing the height of the Cold War, and in fact you see roughly the op­po­site. So this is weirdly re­as­sur­ing. In some sense the Petrov in­ci­dent im­plies that we are slightly safer about nu­clear war.

On the other hand, the MNM effect re­quires lead­ers and in­di­vi­d­u­als to have ac­cess to in­for­ma­tion about the state of the world right now (i.e. how dan­ger­ous are things at the mo­ment). Even in coun­tries with rea­son­ably free flow of in­for­ma­tion this is not a given. If you ac­cept Eliezer Yud­kowksy’s the­sis that click­bait has im­paired our abil­ity to un­der­stand a per­sis­tent, ob­jec­tive ex­ter­nal world then you might be more pes­simistic about the MNM effect go­ing for­ward. Per­haps for this rea­son, we should ex­pect coun­tries with higher so­cial trust, and there­fore more abil­ity for in­di­vi­d­u­als to agree on a con­sen­sus re­al­ity and un­der­stand the level of dan­ger posed, to perform bet­ter. Ja­pan and the coun­tries in North­ern Europe like Den­mark and Swe­den come to mind, and all of them have performed bet­ter than the miti­ga­tion mea­sures em­ployed by their gov­ern­ments would sug­gest.

The prin­ci­ple that I’ve called the Mori­t­uri Nolu­mus Mori effect is defined in terms of the map, not the ter­ri­tory—a place where our pre­dic­tions di­verged from re­al­ity in an eas­ily and con­sis­tently de­scrib­able way—that the short-term re­ac­tion from many gov­ern­ments and in­di­vi­d­u­als was stronger than we ex­pected, whilst ad­vance plan­ning and rea­son­ing was as weak as we ex­pected. The MNM effect may also be a fea­ture of the ter­ri­tory. It may already have a name in the field of so­cial psy­chol­ogy, or sev­eral names. It may be a con­tin­gent arte­fact of lots of lo­cal facts about only our coro­n­avirus re­sponse, though I don’t think that’s plau­si­ble for the rea­sons given above. Either way, I be­lieve that it was an im­por­tant miss­ing piece, prob­a­bly the biggest miss­ing piece, in our early pre­dic­tions and needs to be con­sid­ered fur­ther if we want to re­fine our anal­y­sis of X-risks go­ing for­ward. One of the few up­sides to this catas­tro­phe is that it has pro­vided us with a small-scale test run of some dy­nam­ics that might play out dur­ing a gen­uine catas­trophic or ex­is­ten­tial risk, and we should be sure to ex­ploit that for all its worth.