Honoring Petrov Day on LessWrong, in 2019

Just af­ter mid­night last night, 125 LessWrong users re­ceived the fol­low­ing email.

Sub­ject Line: Honor­ing Petrov Day: I am trust­ing you with the launch codes
Dear {{user­name}},
Every Petrov Day, we prac­tice not de­stroy­ing the world. One par­tic­u­lar way to do this is to prac­tice the virtue of not tak­ing unilat­er­al­ist ac­tion.
It’s difficult to know who can be trusted, but to­day I have se­lected a group of LessWrong users who I think I can rely on in this way. You’ve all been given the op­por­tu­nity to show your­selves ca­pa­ble and trust­wor­thy.
This Petrov Day, be­tween mid­night and mid­night PST, if you, {{user­name}}, en­ter the launch codes be­low on LessWrong, the Front­page will go down for 24 hours.
Per­son­al­ised launch code: {{codes}}
I hope to see you on the other side of this, with our honor in­tact.
Yours, Ben Pace & the LessWrong 2.0 Team
P.S. Here is the on-site an­nounce­ment.

Unilat­er­al­ist Action

As Nick Bostrom has ob­served, so­ciety is mak­ing it cheaper and eas­ier for small groups to end the world. We’re lucky it re­quires ma­jor ini­ti­a­tives to build a nu­clear bomb, and that the world can’t be de­stroyed by putting sand in a microwave.

How­ever, other dan­ger­ous tech­nolo­gies are be­com­ing widely available, es­pe­cially in the do­main of ar­tifi­cial in­tel­li­gence. Only 6 months af­ter OpenAI cre­ated the state-of-the-art lan­guage-mod­el­ling GPT-2, oth­ers cre­ated similarly pow­er­ful ver­sions and re­leased them to the pub­lic. They dis­agreed about the dan­gers, and, be­cause there was noth­ing stop­ping them, moved ahead.

I don’t think this ex­am­ple is at all catas­trophic, but I worry what this sug­gests about the fu­ture, when peo­ple will still have hon­est dis­agree­ments about the con­se­quences of an ac­tion but where those con­se­quences will be much worse.

And hon­est dis­agree­ments will hap­pen. In the 1940s, the great physi­cist Niels Bohr met Pres­i­dent Roo­sevelt and Prime Minister Churchill, to per­suade them to give the in­struc­tions for build­ing the atomic bomb to Rus­sia. He wanted to bring in a new world or­der and es­tab­lish global peace, and thought this would be nec­es­sary—he be­lieved strongly that it would pre­vent arms race dy­nam­ics, if only ev­ery­one just shared their sci­ence. (Churchill did not al­low it.) Our newest tech­nolo­gies tech­nolo­gies do not yet have the bomb’s abil­ity to trans­form the world in min­utes, but I think it’s likely we’ll make pow­er­ful dis­cov­er­ies in the com­ing decades, and that pub­lish­ing those dis­cov­er­ies will not re­quire the per­mis­sion of a pres­i­dent.

And then it will only take one per­son to end the world. Even in a group of well-in­ten­tioned peo­ple, nat­u­ral dis­agree­ments will mean some­one will think that tak­ing a dam­ag­ing ac­tion is ac­tu­ally the cor­rect choice — Nick Bostrom calls this the “unilat­er­al­ist’s curse”. In a world where dan­ger­ous tech­nol­ogy is widely available, the great­est risk is unilat­er­al­ist ac­tion.

Not De­stroy­ing the World

Stanis­lav Petrov once chose not to de­stroy the world.

As a Lieu­tenant Colonel of the Soviet Army, Petrov manned the sys­tem built to de­tect whether the US gov­ern­ment had fired nu­clear weapons on Rus­sia. On Septem­ber 26th, 1983, the sys­tem re­ported mul­ti­ple such at­tacks. Petrov’s job was to re­port this as an at­tack to his su­pe­ri­ors, who would launch a re­tal­i­a­tive nu­clear re­sponse. But in­stead, con­trary to all the ev­i­dence the sys­tems were giv­ing him, he called it in as a false alarm. This later turned out to be cor­rect.

(For a more de­tailed story of how Stanis­lav Petrov saved the world, see the origi­nal LessWrong post by Eliezer, which started the tra­di­tion of Petrov Day.)

Dur­ing the Cold War, many other peo­ple had the abil­ity to end the world—pres­i­dents, gen­er­als, com­man­ders of nu­clear subs from many coun­tries, and so on. For­tu­nately, none of them did. As the num­ber of peo­ple with the abil­ity to end the world in­creases, so too does the stan­dard to which we must hold our­selves. We lived up to our re­spon­si­bil­ities in the cold war, but barely. (The Global Catas­trophic Risks In­sti­tute has com­piled an ex­cel­lent list of 60 close calls.)

Petrov Day

On Petrov Day, we try to live to up to this re­spon­si­bil­ity—we cel­e­brate by not de­stroy­ing the world.

Ray­mond Arnold has sug­gested many ways of ob­serv­ing Petrov Day. You can dis­cuss it with your friends. You can hold a quiet, dig­nified cer­e­mony (for ex­am­ple, with the beau­tiful book­let Jim Bab­cock cre­ated). But you can also play on hard mode: “Dur­ing said cer­e­mony, un­veil a large red but­ton. If any­body presses the but­ton, the cer­e­mony is over. Go home. Do not speak.”

In the com­ments of Ray’s post, Zvi asked the fol­low­ing ques­tion (about a var­i­ant where a cake gets de­stroyed):

I still don’t un­der­stand, in the con­text of the cer­e­mony, what would cause any­one to push the but­ton. Whether or not it would in­cin­er­ate a cake, which would pretty much make you his­tory’s great­est mon­ster.

To which I replied:

The point isn’t that any­one sane would push the but­ton. It’s that we, as a civil­i­sa­tion, are just go­ing around build­ing but­tons (cf. nukes, AGI, etc) and so it’s good prac­tice to put our­selves in the situ­a­tion where any unilat­er­al­ist can de­stroy some­thing we all truly value. When I said the above, I was jus­tify­ing why it was use­ful to have a rit­ual around Petrov Day, not why you would press the but­ton. I can’t think of any good rea­son to press the but­ton, and would be an­gry at any­one who did—they’re just de­creas­ing trust and in­creas­ing fear of unilat­er­al­ists. We still should have a cer­e­mony where we all prac­tice the art of sit­ting to­gether and not press­ing the but­ton.

So this year on LessWrong, I thought we’d build our­selves a big red but­ton. In­stead of mak­ing ev­ery­one go home, this but­ton (which you can find over the front­page map) will shut down the Less Wrong front­page for 24 hours.

Now, this isn’t a but­ton for any­one. I know there are peo­ple with an in­ter­net ac­cess who will hap­pily press but­tons that do bad things. So to­day, I’ve emailed per­son­al­ised launch codes to 125 LessWrong users, for us to prac­tice the art of sit­ting to­gether and not press­ing harm­ful but­tons[1]. If any users do sub­mit a set of launch codes, to­mor­row I’ll pub­lish their user­name, and whose launch codes they were.

Dur­ing Thurs­day 26th Septem­ber, we will see whether the peo­ple with the codes can be trusted to not, unilat­er­ally, de­stroy some­thing valuable.

To all here on LessWrong to­day, I wish you a safe and sta­ble Petrov Day.


[1] I picked the list quickly on Tues­day, mostly leav­ing out users I don’t re­ally know, and a few peo­ple who I thought would press it (e.g. some­one who has said in the past that they would). If this goes well we may do it again next year, with an ex­panded pool or more prin­ci­pled se­lec­tion crite­ria. Though I think this is still a rep­re­sen­ta­tive set—out of the 100+ users with over 1,000 karma who’ve logged in to LessWrong in the past month, the list in­cludes 53% of them.

Added: Fol­low-Up to Petrov Day, 2019.