On Destroying the World
Yesterday I blew up the front page. This was unintentional. I was tricked by one of my friends:
petrov_day_admin_account September 26, 2020 11:26 AM Hello Chris_Leong,
You are part of a smaller group of 30 users who has been selected for the second part of this experiment. In order for the website not to go down, at least 5 of these selected users must enter their codes within 30 minutes of receiving this message, and at least 20 of these users must enter their codes within 6 hours of receiving the message. To keep the site up, please enter your codes as soon as possible. You will be asked to complete a short survey afterwards.
In retrospect, this was quite silly of me. I actually noticed that the account was different from the one that sent the first message, which should have given it away, but the message really did feel legit so I trusted it anyway.
But beyond this, there were further details that should have made the message somewhat suspicious. The first is that this experiment occurred after midnight for San Fransisco. Given that most of the users on this site are based in the US, they wouldn’t have been awake. While they might have specifically chosen users from suitable timezones, it would have made much more sense for them to just wait until more users woke up. Secondly, 20⁄30 users within 6 hours seems a bit high given that users weren’t told in advance that the game was going on, so it’s not clear how many would be available even if they knew.
One thing that greatly surprised me was how much the following comment was misunderstood:
Should I press the button or not? I haven’t pressed the button at the current time as it would be disappointing to people if they received the email, but someone pressed it while they were still asleep.
People read the comment and assumed I was intending the press the button and the only different the trick meant was that it occurred earlier. One of the risks when writing comments quickly is that the meaning might not be very clear at all. I hadn’t actually made up my mind about whether to push the button or not as I was waiting for comments to come in. All I had decided was that I didn’t want the site to blow up while people were still asleep because I thought it’d be less fun for them. That said, I was entirely open to blowing up the site if I thought that the argument for was stronger than the argument against.
Ruby pointed out that I didn’t spend as much thinking about this:
This seems plausible. I do want to note that your received message was timestamped 11:26 (local to you) and the button was pressed at 11:33:30 (The received message said the time limit was 30 minutes.), which doesn’t seems like an abundance of caution and hesitation to blow up the frontpage, as far as I can tell. :P
I saw the email notification almost immediately after it was sent and I thought about it for a bit before deciding that it really just felt legit. I considered messaging the mods, but I assumed they were asleep as it was like 2 am over there. The timestamps indicate that I only spent about seven minutes thinking about it, but it definitely felt longer.
I responded to Ruby with the following comment, which certainly wasn’t the best comment that I’ve ever made.
Well, it was just a game and I had other things to do. Plus I didn’t feel a duty to take it 100% seriously since, as happy as I was to have the chance to participate, I didn’t actually choose to play.
I suppose the thing I should clarify about this comment is, “I didn’t actually choose to play”, as I did kind of choose to play by posting comments asking whether I should press the button on not. What I could have said if I had wanted to be more precise is that at most my commitment to engage was to read the comments that people posted and to take them into consideration. That is, to not waste the time of people who took the effort to reply.
I don’t think I really had a duty to do anything further, including spending the full or most of the half an hour considering the decision. JacobJacob wants to draw a distinction between acting and not acting and I think that’s fair enough for the original version of the game, but as soon as I received the email, the difference between acting and not acting collapsed, and the decision not to act would have been an action in and of itself.
This brings me to Oliver Habryka’s comment:
To be clear, while there is obviously some fun intended in this tradition, I don’t think describing it as “just a game” feels appropriate to me. I do actually really care about people being able to coordinate to not take the site down. It’s an actual hard thing to do that actually is trying to reinforce a bunch of the real and important values that I care about in Petrov day. Of course, I can’t force you to feel a certain way, but like, I do sure feel a pretty high level of disappointment reading this response.
We’ll come to this in a moment, but first I want to address his final sentence: “Like, the email literally said you were chosen to participate because we trusted you to not actually use the codes”. I’ve played lot of role-playing games back in my day and often people write all kinds of things as flavor text. And none of it is meant to be taken literally.
I want to point out a few things in particular. Firstly, the email was sent out to 270 users which from my perspective made it seem that the website was almost guaranteed to go down at some time, with the only question being when (I was aware the game was played last year, but I had no memory of the outcome or the number of users).
Beyond this, the fact that the message said, “Hello Chris_Leong” and that it was sent to 270 users meant that it didn’t really feel like a personal request from Ben Pace. Additionally, note the somewhat jokey tone of the final sentence, “I hope to see you in the dawn of tomorrow, with our honor still intact”. Obviously, someone pressing the button wouldn’t damage the honor or reputation of Less Wrong and so it seemed to indicate that this was just a bit of fun.
But beyond this, I remember when I was a kid and I played games super seriously, while other kids just wanted to have some fun. And I was annoyed with them because I wanted to win, but I felt that others on my team were holding me back. It wasn’t until I was an adult that I realized I was wrong it insist that they had to engage with the game in the same way as me.
Now Habryka is annoyed because he was trying to run a specific experiment and that experiment wasn’t, “Can people who kind of care about the game, but don’t care too much get fooled into taking down the site”. I can understand that, I imagine that this experiment took a lot of time to set up and he was probably looking forward to it for a while.
At the same, the purpose of this experiment wasn’t clear at all. I wasn’t sure if it was having fun, increasing awareness or gaining insight into people’s psychology. I read the email and the post, and the feeling of this “I do actually really care about people being able to coordinate to not take the site down. It’s an actual hard thing to do that actually is trying to reinforce a bunch of the real and important values that I care about in Petrov day” wasn’t really articulated anywhere. And if there was a particular goal, instead of us being supposed to decide for ourselves what the goal was, then maybe it would have made sense to have been clear about it?
- Postmortem to Petrov Day, 2020 by 3 Oct 2020 21:30 UTC; 97 points) (
- 27 Sep 2021 22:16 UTC; 19 points) 's comment on Petrov Day 2021: Mutually Assured Destruction? by (
- A list of Petrov buttons by 26 Oct 2022 20:50 UTC; 19 points) (
- 26 Sep 2021 23:45 UTC; 18 points) 's comment on Clarifying the Petrov Day Exercise by (EA Forum;
- 9 Jan 2022 17:50 UTC; 5 points) 's comment on An Observation of Vavilov Day by (
- 28 Sep 2020 7:39 UTC; 2 points) 's comment on Honoring Petrov Day on LessWrong, in 2020 by (
I feel like this has unintentionally brought us closer to Petrov’s actual experience.
This tradition has so far consisted of leaving the button alone, with no incentives to push it beyond the ubiquitous temptations of pushing buttons and/or trolling. But that is not what happened to Petrov.
Petrov received a message telling him to push the button or the Bad Thing would happen.
Petrov thought the message looked legit, but noticed there were clues that it wasn’t.
Petrov had little time to make the decision.
He went with the clues and we lived. Chris didn’t and we metaphorically died.
We are still unequal to Petrov, for now.
Unintentionally?!?
I am probably not following this as closely as many commenters here, but I 100% assumed it was intentional. It’s just so good!
This is why many of us wondered, when Chris mentioned in the main Petrov Day thread what had happened, whether the message was in fact from the LW mods.
[Epistemic-emotional status: reporting an emotional reaction, which may or may not have correct reasoning; attempting to honor the emotional tone of that reaction without actually wanting to be confrontational about this. Important note: when I say that it feels like people are being shamed, I do not mean think that anyone actually intends to shame people; it’s that I feel that that’s what the tone will communicate to many, intentionally or not. I could be wrong about that too. Again, trying to fairly report the implicit model that one part of me has, without asserting that the model is necessarily fully correct.]
So I notice that there’s a part of me which gets annoyed, borderline upset when I hear this Petrov Day ritual discussed in this manner.
The upset feels similar to what I’ve previously experienced when something that’s obviously a purely symbolic gesture is treated as a Big Important Thing That’s Actually Making A Difference. As it feels like the symbolic thing is distracting from things that actually matter.
That by itself wouldn’t trigger the reaction; the world is full of purely symbolic gestures that are claiming to make a difference, but they mostly haven’t upset me in a long time. Some of the communication around Petrov Day has. I think it’s because of a sense that this idea is being pushed on people-that-I-care-about as something important despite not actually being in accordance to their values, and that there’s social pressure for people to be quiet about it and give in to the social pressure at a cost to their epistemics.
I feel like Oliver’s comment is basically saying “people should have taken this seriously and people who treat this light-heartedly are in the wrong”. It’s spoken from a position of authority, and feels like it’s shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant, as no compelling reason for this ritual actually being significant has ever been presented.
As Chris notes, the communication around this has been contradictory, with some of it being presented as “this is just a game is good fun” and some “this is actually a thing to be taken seriously”. That also feels like a catch-22, where it’s implied that it’s shameworthy to not have taken the game seriously, when people were simultaneously also given the signal that it’s not serious.
I think that to a large extent, I stopped caring about meaningless rituals being pushed within my ingroup because I stopped considering my ingroup to include the kinds of groups that were pushing the rituals. But Chris feels like part of my ingroup, and now I feel upset again because I feel he’s being punished unfairly.
Chris wrote a comment saying this
Before I strong-upvoted it, the comment was down at −11 karma. I read that as Chris being punished, for making a statement that feels like exactly correct to the part of my that feels upset.
Now it may be the case—a more agreeable part of me wants to interject—that this ritual actually is important, and that it should be treated as more than just a game.
But.
If so, I have never seen a particularly strong case being made for it. (Nor even the intention being communicated clearly.) It feels like it was just arbitrarily decided, then imposed on the site from a position of authority, using that position of authority to suppress dissent and strongarm people into acting in accordance with the ritual. There’s a sense of “maybe this is important and maybe not, but you don’t get to just declare it important and require people to act accordingly”.
And that’s why that part of me feels upset about this.
Yeah, I think this is a reasonable reaction, and I really appreciate you going in-depth on your reaction here. And maybe the right call is to basically not have any shared rituals and traditions that have a shared sense of importance, which feels to me like the western secular default.
But, I don’t know, that does leave me feeling pretty empty and sad, and I notice that if I don’t have an active and strong culture around me, that I just default to whatever other random culture around me does have any content, even if I don’t really like the ideas of that culture, and it feels like a pretty major loss to me. I do think that culture is really important, and shared rituals and traditions and games like these feel like how you actually build a culture that has any substantial content. And I really like Petrov Day. I consider it and Solstice to be the two primary holidays we have that we get to shape to reinforce our shared values and ideals, and want us to make use of them.
Like, I do think it’s important that we try our best to only send the invites to people who are up for taking this seriously, and it should be easy to opt-out. I think we should improve the communication to make it clearer that “yes, I do think this is a serious challenge. It’s easy for you to opt out by just ignoring this message, but please don’t destroy what I care about because you just want to have some fun. Building trust that I can have things I care about without other people destroying them just for fun, or because they dislike me or want to actively hurt me, is like half of the point of the whole exercise”.
Like, I don’t want this to be a somber serious thing. I think traditions need some fun to stay alive and not be super draining for everyone. But I also don’t want it to just be a thing that people don’t take seriously at all. Getting the communication for that right is difficult. Like, I want it to be serious play. A bunch of people who do actually try to do the right thing and win, but with an understanding that yes, this isn’t fully the real thing, this is practice, and the stakes are lower, but also not nothing, because we are doing this because we care about the bigger thing. It feels to me that most competitive sports get a substantial part of this tone right, and wargaming also has some of this feeling.
Like, I was really proud when last year we successfully didn’t blow up the site. I was proud when people considered blowing it up, and then people talked about it, and made arguments, and overall the, what seemed to me, right considerations prevailed. I think it was a pretty real and substantial achievement, and it gave me a sense of trust that everyone we sent the codes to shared at least some values and virtues with me, or at the very least wasn’t trying to actively hurt me. That trust was worth a lot. I would likely give you $5k of my own money for that trust.
And like, maybe other people don’t care much about that trust, but overall, I feel like I am in a world where trust is very scarce. Most people don’t take anything seriously and live life in what feels to me like a postmodernist careless perspective on the world, or alternatively are mindkilled and overwhelmed with values I don’t share and feel scary to me. And I when I think of a world where we successfully have 500 people over the course of 10 years, who every year successfully not blow up the site, and use it as an opportunity to remember the fragility of civilization, then that makes me pretty happy, and makes me more excited to invest in LessWrong and all the associated communities.
And like, I don’t know, what is authority for if not for coordinating exactly this kind of cultural institution? Of course, we should lose some of that authority as we do things that people don’t like, and if we repeatedly do stuff that lots of people don’t like, the authority we have should be taken away from us. But, I think, overall, my sense is that the community is happier if we use our authority to sometimes try to establish traditions like this. It is of course important that what we do isn’t a catch-22, nobody wants that. But I don’t think what happened here is a catch-22. Some of what happened here is a misunderstanding, and some of what happened here is simply the cost that you sometimes pay if you try to send a costly signal, but can’t quite get it right.
Like, I do think we should have done better at communicating the degree of seriousness of this thing. I have some ideas of how we can do that better in the future. I also sure really think Chris should have done a better job at reading the intention that was behind the email, and I did lose some trust in him and others that they will take things I care about seriously, and that they won’t just fall prey to relatively low-effort phishing attacks when it actually matters that they don’t.
Hey, I’m relieved and grateful that you took this so well. :) I hesitated for a while before posting my comment. I get that this ritual was important for you and didn’t want to disrespect that; probably also didn’t speak up last year because I wasn’t sure I could communicate it in a good way.
I totally get the desire for rituals, and think it’s an important one; I haven’t been to a Solstice but I appreciate what they’re doing. I also don’t have a problem with them, maybe because they don’t feel like they are trying to claim anything that they’re not.
Generally most of my problem with this ritual was a) some aspects of its execution, such as the communication, which is fixable, and b) the feeling that it’s actually not very analogous to the dilemma it’s trying to be symbolic for, and which it claims to be training people in. (I said a few words about that in the final paragraph of my response to lionhearted.) I think that if it really felt to me like it was teaching people to be more trustworthy and coordinate better in situations-like-the-one-Petrov faced, then I’d probably be very happy to have it around. I just don’t feel like it’s there… yet. :)
Yeah, I do think we are definitely still figuring out a bunch of rough edges around this tradition.
I do think that what we set up is reasonably analogous of what I think happened to Petrov, and the aims I have for it, though of course I would think that, but here is maybe one more attempt at synthesizing what values I care about reinforcing with Petrov Day:
Remembering that humanity is fragile, and that we have come close to destruction in the past, and will likely come close to destruction in the future.
This feels like it really straightforwardly resonates with the setup. You need to have some chance that things will go wrong to create a real sense of tension, but of course you don’t actually want to have stakes that are so high that they destroy lots of value if you mess it up, in particular while you are still dealing with a lot of uncertainty about the setup.
Practicing the virtue of not taking unilateral action and being mindful that your actions can have large negative consequences, and that you will act responsibly with the power you are given
I think this is just a really valuable virtue for a community of people to have, and I do think it’s what distinguished Petrov from most people in similar reference classes to him. Like, as I said in the other comment, he could have just been a bureaucrat who didn’t care about his job, didn’t pay much attention to what was happening, and just accidentally contributed to destroying the world. And similarly, I want the people around me to take responsibility for their actions, in particular if they have a large potential downside.
I think of these two as the core virtues of Petrov day, and I think our current ritual does a pretty decent job at reinforcing them. Like, last year when the LessWrong frontpage had a chance to go down any minute, it really felt very analogous to how I imagine people must have felt like during the cold war, where a catastrophe could have happened any minute, and suddenly destroy something I really care about. This year, the way it failed did actually really make me think that there were probably at least a couple of nearby counterfactual worlds where the cold war happened because some opsec protocols weren’t good enough, and some third party in the cold war just did some social engineering to cause a major war between the U.S. and Russia, hoping that they would end up on top in the resulting chaos.
On a meta level, I think next year’s email and post should link pretty prominently to a post just titled “Why LessWrong Petrov Day?” that explains the reasoning here pretty straightforwardly.
This sounds like it’s pretty well captured by current Petrov Day ritual, yeah, though I feel like it only being the front page rather than all of LW makes it feel much less serious.
Doesn’t Petrov’s choice actually get closer to taking than not taking unilateral action, though? The current ritual captures “think carefully about your actions”, yes, but as I understand it Petrov was supposed to report a missile launch to his superiors. who could in principle also have used their judgment to dismiss it as a false alarm.
He did the right choice, no doubt, but it feels weird to use “saw an event that could have led to the end of the world, made a choice that involved going against his standing orders and the previous planning that many others had participated in, ultimately making the decision purely himself rather than communicating it to the people with the pre-designated authority to deal with it” as a symbol of coordination and avoiding unilateral action.
Sorry, after thinking about this, I basically think that “unilateral action” is just a confusing choice of words. Let’s replace it with “being given substantial purely destructive power, and wielding that power responsibly,”, because I think while there was a substantial unilateral component to the cold war, I don’t think Petrov’s choice in particular was that reliant on unilateral considerations.
I mean, I think taking all of LessWrong down would be a bit of a dick move. Like, the frontpage is what matters most to the people who participate and is a resource that feels fair and reasonably to destroy, because it being down mostly just costs the people who participate in the ritual.
But I feel like as a developer on LessWrong I have a pretty serious responsibility to be a good shepherd of content, and to make sure that you can reliably link to LessWrong content, and that you can reliably read the sequences, without it breaking. Most of the people who read that content aren’t regular users, they are people who got linked here from some other blogpost on the internet, and I don’t want to externalize our bad decisions into giving them a bad experience.
I think in general, I wouldn’t want to run rituals like this that randomly damage some public infrastructure. Like, I wouldn’t want to make it so that when someone presses a button, we barricade a random road in Berkeley. Making all the content on LessWrong inaccessible feels similar to that. It’s not my right to remove people’s access to that.
I don’t have a strong opinion on how serious Petrov Day should be. Just that if you wanted it to be taken more seriously then it should have been set up differently.
Great comment, naturally. I appreciate your epistemic status quite a bit.
I think I want to respond to the idea that it’s contradictory and bad to signal that the Petrov Day button is serious and signal that it’s fun. A few examples:
HPMOR is a book about growing up, failure, and death. It’s also hilarious and riveting.
Unsong is 50% puns. It also contains a chapter describing hell and torture in some detail.
Embedded Agency is research done downstream of the potential for advanced optimizers to lead to an existential catastrophe for humans. It’s also a cute and colorful cartoon.
Previously warring countries often come together and have their sports teams play. It really matters that they don’t cheat and play honorably, even if it’s “fun” and “play”. It’s a game, but it’s not “just a game”.
Some animals sheath their claws for dominance fights, where the losing player loses real status but isn’t physically harmed. Again, it’s a game, and it’s in some ways play. And it’s also serious.
I’m here trying to build a community around the art of rationality. We also do an April Fools’ joke every year, like that time we made everyone’s font size proportional to their total karma for a day.
People have birthday parties, and their friends show up. And it means something for your friends to show up. It’s a party, and it’s also a true signal of friendship and being there for your friend, and you can be disappointed in them not showing up, even if it’s ‘fun’ and ‘just a party’.
I don’t think it’s contradictory to care about something deeply and to be playful with it. As with Feynman and physics, Eliezer and HPMOR, Scott and Unsong, Abram/Scott and Embedded Agency; also, LessWrong and Petrov Day.
Relatedly, (part of) you said that sometimes I signaled that this was serious, and sometimes I signaled that it was ‘just a game’. I think this is incorrect. I signaled it was playful, but never unserious, never ‘just’ or ‘merely’ a game. Everything I wrote was about trust, honor, extinction, gratitude, mourning, and being Beyond the Reach of God. I didn’t write anything that suggested you should consider pressing the button, or was secretly winking at the audience. Taking down the site is symbolic, but that doesn’t mean it’s ironic or a joke, they’re totally different. We’re symbolizing world-ending destructive technology, with a much lesser but still destructive technology. I care about this tradition a lot, and it was part of what was involved and in everything I wrote.
I feel like it takes a very cynical prior to read at everything I wrote, consider that I actually cared, then go “Yeah, he probably doesn’t mean this, why would someone actually care about this, he’s probably joking?” I don’t think anyone has that prior… I think they more have a prior that people rarely actually care about things, and so when they look at something that was meant straightforwardly and un-ironically, the hypothesis isn’t even brought up to attention.
But a lot of people got it. Most, I think. I got a bunch of short response messages to getting the codes (and some long ones) saying things like “I’m honored to be entrusted with the launch codes.” and “Roger that, general. I won’t let you down!” and “I’m honored that you gave me the opportunity” and “Awesome, thanks! I love the warm glow of not burning the commons.” I love getting these messages. And I love that it worked out last year.
It’s a distinct argument to say that I was unsuccessful at assuring that was communicated before someone took action, which this is some evidence of. Like, last year someone visited the site, pressed the button, and entered a string of zeros then hit ‘submit’, before finding out what the button even did, and without reading the announcement post. (We responded to that this year by having two sentences right underneath the button saying what was happening.)
I take seriously the charge that users like Chris would’ve gotten it if I had re-written the email and post in some ways, and I will definitely user-test it more next year. You say it’d be good to write down the case for the tradition; I can also do that, write a post called “Why the Petrov Day Big Red Button?”, and link to it from everywhere next year.
But… well, there’s more to say, but I have to go for now. I’ll add that it was my responsibility to pick the people and write the announcements. I entrusted 125 people with codes last year, and succeeded. I tried to do 270 this year, and I failed. I’m writing a postmortem, and I will work hard to ensure the site doesn’t go down next year.
We often distinguish between safety critical and non-safety critical components. The latter make up about 95% of components in my business and in general the thing we care most about is average performance.
In safety critical components we care about the worst component (material / manufacturing defect etc.) in e.g. 1,000,000. Otherwise >1 in 1,000,000 brakes fail and the vehicle runs someone over or drives into a canal.
The examples that you give of jokey but serious things are almost all non-safety critical things (except the dominance contest but I think that’s quite a different example). If I miss that embedded agency is about something serious then that doesn’t really matter—someone who makes that mistake is probably not really who it is important to make understand. The overall effect of the series is the most important thing.
My impression is that the message you sent is great for average performance (and that the most natural way to read it is as you intended) but that it isn’t optimised for communicating with the biggest exception in 270. The person who shares the least common knowledge about the ritual or reads the message the fastest or has the prior you mention or a prior that the admins sometimes do pure jokes (e.g. April Fools) or whatever—that person is really the person you need to be writing for. That most people understood it correctly is largely irrelevant.
I feel like this is a huge lesson that this experience hammered home for me.
(The message changed slightly from last year to this—one section I particularly note is:
in 2019 was changed in 2020 to:
I’d be curious to know why this was changed as the former seems better optimised for setting expectations.)
This is a solid point. And I’m glad to hear you learned a lot from this, I think many of us are (I know I am). I still think something is being missed, let me say what that is.
Like, there’s the part where I try to correctly pick people and communicate with them, where I take the effort to ensure they understand what’s going on and don’t take destructive action. I will do more user testing. I will check my writing more with friends and colleagues. I will spend more time reading the comments and posts of users I’m planning to give codes to. Perhaps I will make it take a little work to get your codes, like filling out a brief form that says “Yes, I understand this is a tradition about not taking massively destructive action” and “Yes, I understand the Frontpage will literally be unavailable for 24 hours if I submit the codes” and “Yes, I understand that the LW team will never ask me to submit my codes” before you get the codes. Certainly I will update the general algorithm I used to select people this year, which I realize in retrospect was fatally flawed (post in-progress on that). There’s lots to do better here on my part, certainly. I will do better.
There is also the part where, as a person in the world and as a user on LW, you decide what principles you have and what responsibilities you can take on. How hard do you try to generally be the sort of person who does not needlessly destroy things? Do you want to be the sort of person who can be entrusted with destructive power? (This is often a prerequisite for being trusted with any significant power.) Regardless of the reasoning, we gave everyone a destructive button for a day. If you’re the sort of person who wants to be trusted in this way, both on Petrov Day and with matters even more important, you should not use it, not joke about using it, not try to use it as leverage for personal gain, and so on and so forth.
Many people don’t take this sort of responsibility. Feynman, for all his other virtues that I admire, has this quote about how he actively tries to take no social responsibility, and then he has another quote about realizing the horror of what he had (unwittingly) done in developing the atom bomb. For all his virtues, Feynman doesn’t have the virtue of Petrov. And the commenters under Chris’s Facebook post, on hearing that he had the power to take down a public utility for 24 hours, started trying to find reasons for him to do so. Maybe this is playing on hard mode because people don’t really believe the internet exists, so the damage of taking down the Frontpage that around 2,500 people use in a day is very abstract. But also that’s still kind of the whole point. If you are given the ability to do a lot of damage in a way that feels abstract, how seriously do you take it? Human extinction is abstract, it is hard to think straight about it or connect to it emotionally. The Cold War had a lot of people not taking responsibility for developing and commanding intensely destructive power, and we celebrate Petrov as someone who did.
Here’s another way of looking at it. Below is a PM I got from a lurker (someone I know IRL, I think has read the sequences, but who has like 5 comments ever):
My guess is this user would pay like $10 to not have the site be down to him randomly for one day of the year. (I agree there are secondary effects of it happening once, but I think in general the user will continue to be basically unhappy with this experience.) I think many users who came to use the site and found the frontpage down would have paid a couple of dollars to not have this experience. On most days we get between 2,000 and 3,000 unique visitors to the Frontpage (it’s higher for the site as a whole, around 10k). In direct effects, I expect users would have on-net happily paid something around $3,000 to not have had the site suddenly go down.
(Naturally there are other costs. Last year we had a big spike in the metrics on this day, but not this year. And last year several users said they’d be willing to pay a few hundred dollars apiece to not take the site down for Petrov Day, which is more likely to be the main cost. But let’s stick with the direct effects and just call it $3,000 of community resources.)
One of Chris’s friends tricked him into burning $3k of community resources and Chris said “Well done”, “Well played!” and “if you wanted it to be taken more seriously then it should have been set up differently”. I think this isn’t taking responsibility for the actual damage caused.
(This doesn’t devalue the other contributions Chris makes to LW and EA, like making open threads and FB groups and writing posts and so on, all of which I’m grateful for.)
I’m not saying everyone did the above Fermi estimate. But I think a lot more people have built up the sense of “I’m being given responsibility for the commons, and I am the sort of person who won’t play with that, even if I think it’s silly or bad that I’ve been given this responsibility, or don’t have quite a clear sense of what scale of commons resources this is. (It’s just the Frontpage? It’s just 24 hours? How bad is it really?) I still won’t mess with it.”.
This is part of why I’m not as sold that the minutiae of phrasing was the key crux on my part that caused the site to go down, and I think it was much more in the selection criteria for people. (As I say, post-mortem incoming.)
I really don’t see the frontpage being down for a day as that bad. I guess that a lot of us (myself included!) spend too much time on the Internet, so maybe very occasionally not being able to access a particular site is a good thing?
Anyway, I definitely would have been more careful if I knew that this was something people cared about so much.
Do you disagree with the Fermi estimate about how willing-to-pay people would be? (And more than that, do you think most websites should be randomly down for a day – that this would be a better state of affairs? I don’t expect you do.)
I’m glad to hear you would have been more careful if you knew people cared so much.
I think it’s reasonable Fermi estimate. That said, people would be willing to pay a lot more for Facebook. Doesn’t mean it’s more valuable.
“Do you think most websites should be randomly down for a day?”—well, it’s not just down for no reason. It’d also issuing a reminder of the importance of existential risk. And then people would be able to read about who took it down and possibly why afterwards. And there could be significant utility there.
But I guess my true rejection is that I saw it as just a game and I assumed that you wouldn’t set up such a game unless you judged the cost to be insignificant. Which seemed inline with just taking down the frontpage for a day. And, I didn’t see any reason to double check what I presumed your judgement was here.
There’s a lot of stuff I agree with here and some stuff I’d push back on but probably worth waiting for the post-mortem before going deeper.
That’s actually a pretty big difference.
Hey—to preface—obviously I’m a great admirer of yours Kaj and I’ve been grateful to learn a lot from you, particularly in some of the exceptional research papers you’ve shared with me.
With that said, of course your emotions are your own but in terms of group ethics and standards, I’m very much in disagreement.
On the one hand, you’re totally right. On the other hand, basically the entire world is made up of abstractions along these lines. What can the Supreme Court opinion in Marbury vs Madison be recognized as other than a purely symbolic gesture? Madison wasn’t going to deliver the commissions, Justice Marshall (no relation) knew that for sure, and he made a largely symbolic gesture in how he navigated the thing. It had no practical importance for a long time but now forms one of the foundations of American jurisprudence effecting, indirectly, billions of lives. But at the time, if you dig into the history, it really was largely symbolic at the time.
The world is built out of all sorts of abstract symbolism and intersubjective convention.
Canonical reply is this one:
https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/7FzD7pNm9X68Gp5ZC
(“Canonical” was intentionally chosen, incidentally.)
https://www.lesswrong.com/posts/tscc3e5eujrsEeFN4/well-kept-gardens-die-by-pacifism
From Well-Kept Gardens:
Honestly, for anything that wasn’t clearly egregiously wrong, I’d support the leadership team on here even if my feelings ran in a different direction. Like, leadership is hard. Really really really hard. If there was something I didn’t believe in, I’d just quietly opt out.
Now, I fully understand I’m in the minority on this position — but I’m against both ‘every interpretation is valid’ type thinking (why would every interpretation be valid as it relates to a group activity where your behavior effects the whole group?).
Likewise, pushing back against “shaming people whose main sin is that they aren’t particularly persuaded by this ritual actually being significant” — isn’t that actually both good and necessary if we want to be able to coordinate and actually solve problems?
There’s a dozen or so Yudkowsky citations about this. Here’s another:
https://www.lesswrong.com/posts/KsHmn6iJAEr9bACQW/bayesians-vs-barbarians
And finally,
I made that case last year extensively:
https://www.lesswrong.com/posts/vvzfFcbmKgEsDBRHh/honoring-petrov-day-on-lesswrong-in-2019?commentId=ZZ87dbYiGDu6uMtF8
I even did, like, math and stuff. The “shut up and multiply” thing.
Long story short — I think shared trust and demonstrated cooperation are super valuable, good leadership is incredibly underappreciated, and whimsical defection is really bad.
Again though — all written respectfully, etc etc, and I know I’m in the minority position here in terms of many subjective personal values, especially harm/care and seriousness/fun.
Finally, it’s undoubtedly true my estimate of the potential utility of building out a base of successfully navigated low-stakes cooperative endeavors is undoubtedly multiple orders of magnitude higher than others. I put the dollar-value of that as, actually, pretty high. Reasonable minds can differ on many of these points, but that’s my logic.
Thanks for engaging :) My upset part feels much calmer now that it has been spoken for, so I’m actually pretty chill about this right now. You’ve had a lot of stuff that I’ve gotten value from, too.
But note also that that post contains a lengthy excerpt about how the “Dark Side” descends into cultishness and insanity in situations where the word of leaders is accepted without question. That was clearly also depicted as the opposite failure mode.
I agree that rationalists don’t cooperate enough, and that often just offer criticism when it’s not warranted. But… it feels like a Fully General Counterargument if you take to that the point of “no coordination may be criticized, ever, including situation where people are arguably being shamed for having good epistemics”. That sounds like this bit from the post:
Re: Well-Kept Gardens—again, that feels like a Fully General Counterargument. Yes, certainly there should be moderator action when necessary… like, I am on the mod team, I have seen discussions about banning users etc. and all of that’s been fine to me.
But we’re not even talking about banning users here. This isn’t about keeping the garden, it’s about one particular ritual being picked to be important. Does Well-Kept Gardens imply that everything the admins do should be treated as correct?
Back when I banned Eugine Nier, there were people who disagreed with that decision… but if they didn’t find my argument for the ban to be particularly persuasive, it never even crossed my mind that I should shame them because I had made an argument that didn’t compel them. They disagreed, the coordination necessary for saving the site still happened, no reason not to let them disagree.
So I read this to be a central premise that much of the rest of your comment builds on:
Which to me feels like it’s assuming thing that I was asking someone to prove. Yes, Ben feels that the button was symbolic and analogous to not nuking another country. But it does not feel at all analogous to me; I feel like the ritual is picking a surface-level aspect of what happened to Petrov (something like the general idea of “you should not push the red button, doing so would have bad consequences”), and that’s about it when it comes to having a resemblance to it. The outcome matrix and psychological situation for the users is something completely different than what it was for Petrov. If the ritual actually did have a clear structural resemblance to Petrov’s dilemma, then I would have much less of a problem with it. But as it is, it does not feel like it’s training the site’s users in the thing that it claims to be training them in.
Good points.
I’ll review and think more carefully later — out at dinner with a friend now — but my quick thought is that the proper venue, time, and place for expressing discontent with a cooperative community project is probably afterwards, possibly beforehand, and certainly not during… I don’t believe in immunity from criticism, obviously, but I am against defection when one doesn’t agree with a choice of norms.
That’s the quick take, will review more closely later.
I mean, this is a fine judgement to make, but also a straightforwardly wrong one. Last year we had ~150 people, and the site did not go down, with many people saying that we really have to add more incentives if we want to have any substantial chance of the site going down. I do think it’s a pretty understandable mistake to make, but also one that is actually really important to avoid in real-life unilateralist situations.
Of course it damaged our reputation! How could it not have? Being able to coordinate on this is a pretty substantial achievement, and failing on this is a pretty straightforwardly sad thing to happen. I definitely lost a good amount of trust in LessWrong, and I know of at least 10 other people who independently expressed similar things. Again, it’s an understandable mistake to make, but also straightforwardly a retrospectively wrong statement.
To be clear, I think in real-life situations, people taking the consequences of their actions not seriously and treating things as just a game to be played is a serious path towards real-life risks! I don’t think you destroyed the setup for this experiment at all. Indeed, someone not thinking for very long about the consequences of their actions, and taking an action with pretty serious consequences out of carelessness is one of the primary ways in which I expected the frontpage to get nuked, and was an intentional part of the test that I wanted to perform. Indeed, usually norms deteriorate by people disassociating from them, and saying that they never felt they were real in the first place, and brushing things off as inconsequential.
To clarify this some more, in a bit of a rambling way: different people have different values. While it is obvious to us in this community that destroying civilization via a nuclear war is pretty bad, there are many people who when asked “if you could wipe out humanity with the press of a button” would happily go and press it, especially if they didn’t think much about it, because they have the cached belief that civilization is probably overall bad and that life would be better off without humanity. Or many people believe in an afterlife and that the apocalypse would overall cause there to be less sin, or whatever.
I assign substantial probability to the world being destroyed not by someone who wants to destroy the world, but by someone who just doesn’t really think that their actions will have substantial consequences. Like Petrov could have just been a normal bureaucrat, doing his job, following the protocols that were set out by him, and it’s really not hard to imagine a Petrov who just didn’t really care about his job. Who realized in the abstract that nuclear war was a thing, but didn’t really care about it viscerally, and when given the read from the instruments, just didn’t think about it very hard and forwarded the signals to his superiors. That’s how bad things happen. The most likely world in which civilization ended because Petrov didn’t intervene, seems to me to be one where Petrov’s attitude was overall pretty similar to your attitude here. That doesn’t mean your attitude is wrong, I believe that you would actually care about the real case of the nuclear weapons, and am not at all saying that you specifically wouldn’t have done the right thing if the real deal was on the line, but that the reference class of the reason why you didn’t do the right thing here (by my lights) is pretty representative of the reference class by which I expect things to go wrong in reality.
Communicating and coordinating on shared priorities and values is really hard. It’s a way lots of things break. In this case, we clearly failed at that. But also, that’s part of the challenge of building a real and important thing. In real-life, you don’t get to assume that everyone who is working with you actually really cares about avoiding nuclear war with your enemies. You don’t get to assume that everyone has a shared understanding of humanity being really important to preserve, and that being cautious with humanity’s future is of utmost importance. Most people don’t viscerally believe those statements, so you can’t just build coordination on that assumption, and if you do it anyways, I think things will fail in pretty analogous ways to how they failed on Saturday.
I think this is only true conditional on the Petrov Day LW Button being a Serious Thing. But the whole question is whether or not we should consider it a Serious Thing in the first place. Outsiders likely won’t, they’ll just see it as a game.
More generally, the Schelling choice is rabbit not stag—where Rabbit is “don’t take this seriously” and Stag is “do take this seriously”. By putting this thing online and expecting everyone to take it seriously, but without providing a really solid justification for why they should, you’re choosing Stag without prior coordination, which is generally a bad strategy.
(I also endorse Kaj and Neel’s comments below).
Hmm, I do think this is right. But I do think the payoff matrix here is pretty asymmetric. Like, I think it’s obvious that on-net, given reasonable levels of ambiguity, you will lose some reputation. There is a question of how much, but I think it’s pretty reasonable to assume that you will lose some good amount, because at least some fraction of people will take it seriously.
Like, I do think that it’s fine to push back and say that it should just be a game, and that the people who are taking it seriously are wrong, but as a statement about social reality, predicting that it will not cost you reputation just seems like a wrong prediction.
Like, I am fine with the statement that it shouldn’t cost you reputation. But saying that it won’t cost you reputation, feels pretty wrong.
I think we lose some reputation if people think that we are unable to choose Stag even in Serious Situations. But the main thing that signals to outsiders that this is a Serious Situation instead of a fun game is the disappointed reactions after someone chooses Rabbit. (By default outsiders are much less likely than insiders to think of this sort of thing as serious, and it was already ambiguous enough that many insiders didn’t think of it as serious). If the community reaction was more like “What a great learning experience” and “This is a super interesting outcome” then I doubt there’d be a significant reputational cost. I’d estimate that the cost in weirdness points of running this event in the first place is about an order of magnitude higher.
An analogy: suppose the military practises a war game and sometimes fails to achieve its goal. I don’t think this means they lose reputation. In fact, for certain classes of games, you lose more reputation by always succeeding in your goal, because that means that the goals are rigged. Same here: maybe the LW team sent out the invitations in such a way that they were very confident someone would push the button, or maybe in a way where they were very confident nobody would; I can’t tell from the outside.
Yeah, to be clear, I do think it is actually a valuable signal to have failed at the Petrov Day goal at least once, because it signals pretty credibly that things are not rigged, and failure is possible.
I do also think that if you want your war game to be taken seriously as a sign of your competence, it’s important that both you and the people you were war-gaming against were playing seriously. This doesn’t mean that the war-game had to be a “Serious situation”, but it does mean that your soldiers shouldn’t have just gone “lol, it’s just a game” and started playing cards or something because they got bored.
Like, sure, we could make this just a fun game, which would cause us to also not have to be worried about reputational risks, but I don’t see much value in the version of this that is just a fun game, with no serious component. I am not super confident about the right balance of seriousness and fun, but I am pretty confident that a world where nobody took this seriously just doesn’t seem very interesting to me. It doesn’t allow me to build any real trust with anyone else, and feels like it deteroriates the real and important lessons we can learn from Petrov Day.
“Of course it damaged our reputation! How could it not have?”—I suppose it might if it’s a serious exercise and a lot more people seem to interpet it that way than I expected
Thanks for writing this! It seemed like people were being unwarrantedly unfair to you in that thread.
My personal experience was getting the email from Ben, and this being the first I’d ever heard about LessWrong’s approach to Petrov Day. And I somewhat considered pressing the button for the entertainment value, until I read the comments on the 2019 thread and got a sense of how seriously people took it.
I think it’s completely reasonable to not have gotten that cultural context from the information available, and so not to have taken the whole thing super seriously.
And personally I found it fairly entertaining/education how all of this turned out (though it’s definitely sad for all the Pacific time people who were asleep throughout the whole thing :( )
EDIT: Just wanted to add that, now I have the cultural context, I think this was all an awesome celebration and I’m flattered to have been invited to be a part of it! My main critique was that I think it’s extremely reasonable for Chris not to have had the relevant context, but many of those commenting seem to have taken this background context as a given, since it’s clear to them.
I’m genuinely confused about the “pressing the button for entertainment value”.
The email contained sentences like:
And no sentences playfully inviting button-pressing.
Maybe I can’t unsee the cultural context I already had. But I still imagine that after receiving that email, I’d feel pretty bad/worried about pressing.
I resonate with basically all of this from Chris’ post
Trying to introspect a bit more, I think that unseeing the cultural context is hard, and that that context massively affects your priors of how to interpret something like this. My first reaction was that the email was a joke. Then, realising it wasn’t a joke, being confused by why I’d been sent it (the email began Dear Neel_Nanda_1 , not Dear Neel, which made it seem less like I’d been specially chosen). And then, realising that they’d actually changed the Front Page, and done this before, being really entertained at the idea of celebrating Petrov Day in this way. But it felt like “this is a fun, slightly over the top way of celebrating, and we want to see interesting and fun things happen”.
I think my priors are so far from people taking something as minor as “the LW frontpage goes down for a day” seriously, that it took me reading the thread under jefftk last year about selling his launch codes for counterfactual donations, and seeing people genuinely debate “is this worth more than $1.6K” to realise that people took the symbolic value really seriously. (And I’m still pretty confused by this—if I had read about Petrov Day 2019, and saw that someone blew up the front page for a large donation to AMF, that would probably marginally raise my opinion of LessWrong. And I utterly do not understand people who would price as over $10K, let alone $1m)
Things that I think would have changed my intuitive framing:
Having the email drop out of “RPG flavour text mode” and be explicit about the cultural context and how seriously people took it
Having the downside be actually, meaningfully high (eg, LW the website going down for a month) ((though I think this is net bad for the obvious reasons)). As is, it didn’t feel like something to be taken seriously, because the actual stakes were low.
Being given context and invited before Petrov day, and needing to take some agency to accept. I think this would have made the notion of “you are being invited and trusted clearer”. I was surprised by receiving the email and don’t see myself as a notable LW contributor, and assumed eg this was automatically sent to the 270 most recent posters, or highest karma users or something, rather than having been hand-picked by Ben
(In writing all of this, I feel like I’m being unfair to Ben/implying all of this should have been obvious to you guys. That’s not at all my intent, and I hope you take this in the spirit of “an attempt to narrate my internal experience, that might help with orchestrating future things”)
Idk, hope all that helped. This kind of thing is far outside my standard conception of how people and communities work, and I’m not used to people taking symbols this seriously. And I’m surprised by how obvious this all feels to people with the cultural context
This comment helped me understand better what it looks like without the ‘cultural context’, thx.
Just to reply to one particular thing:
Do you agree that the literal monetary value of the site being down for a day is (likely) greater than that? Never mind the symbolism, there’s just like two thousand people who visit the Frontpage in a day, around half of whom might pay something in the range of $1-$10 for the site not to be randomly down on them for a day?
I do not agree that the monetary value of this intervention is anywhere near that.
I didn’t notice—I don’t look at the front page, just using my bookmark to /allPosts. Many use GreaterWrong or RSS, and would be unaffected. Are there logs on how many actual visitors saw the front page down and did not hit any other pages until it was back up?
I’m a pretty heavy site user, and I would not pay $1 to have the site up a few hours or a day earlier in case of an outage. I’d likely pay on the order of $0.10-0.25/day on an annual basis if asked (and if it were a registered charity where I understood how my donation would be used), but having a day or two of downtime is just fine with me.
I’d especially not pay $1 to have the site be up sooner in case of a ritual/demonstration that is intentionally created by a site admin. If they think having the site down for a bit is a positive thing (indicated by the fact they wrote the code to do it), I defer to their wisdom.
Right. I’ll briefly reply to each point:
You don’t look at the LW Frontpage, and neither do GW users or RSS users. This means you and they are outside the set of 2,000 daily visitors, so your lack of inconvenience is not evidence about theirs.
(I don’t have any logs, may look into getting some. As we don’t have that info our uncertainty around that should be factored into the estimate.)
You not wanting to pay $1 if the site was down is indeed a datapoint. I think many people would be fine with an outage. (I still think many would find it irritating.)
I understand that you especially wouldn’t in the case of the symbolism. I’m just trying to pin down the object level effects, to understand what was at stake before counting in all the symbolism.
Overall I’m not certain, it’s plausible the number is lower...
I don’t know how many of the 2000 would do the same thing but switching to GW for the day was fairly obvious to me. On the other hand I use GW on and off so this maybe gave me an advantage but I think the post on surviving the outage suggested doing that too. Short of checking GW traffic I guess it’s hard to know how many did this.
It is noteworthy that I think the sort of person who would bother to pay a dollar to keep the site up is also the sort of person who disproportionately might use greaterwrong (or, for that matter, the /allPosts page). The frontpage gets a lot of views but I think most of them are people who aren’t using LessWrong that seriously.
I said earlier to Ben I thought the $3k number was at least plausible and seemed within an order of magnitude of right. But thinking more I do suspect it’s on the lower end of that order of magnitude I think there’s only a few hundred users for whom the LessWrong frontpage is actually enough-better than whatever else they might be doing that day that they might pay a dollar.
GW had about 40 additional users show up on that day (which corresponds to roughly 35% traffic increase)
If the entire site (not just the front page) was down, then I might on some days pay $1, if I was writing something where I wanted to cite/reference an older article. Otherwise, if I knew it would only last for a day, I would just wait it out.
Interesting. My intuition was “24 hours isn’t a long time, and it’s just the front page, people can surely come back later”. But while that’s a small inconvenience, $1 worth of inconvenience sounds plausible. So yeah, fair point! $1-10k actually seems like a fair value for this, thanks
EDIT: Reading the other comments on that point, it seems reasonable that LessWrong power users are best able to work around the outage, and the people who’d be most inconvenienced. And I expect most of those people to not know about GW (what is GreaterWrong anyway?), but this to correlate with caring less about the existence of LW. So I guess I’d lower the estimate a bit
https://greaterwrong.com is an alternate interface to LessWrong, implemented by… I think Clone of Saturn does most of the coding and Said Achmiz does most of the design work?
Same content, different design, slightly different set of features. (E.g. no karma change notification, no voting on tags, but comment navigation is improved.) I tend to use it over LW because it’s faster.
You can generally just replace lesswrong with greaterwrong in a URL.
Oh, thanks! That sounds really useful when LW is being slow on mobile
As someone who didn’t receive the codes, but read the email on Honoring Petrov Day, I also got the sense it wasn’t too serious. The thing that would most give me pause is “a resource thousands of people view every day”.
I’m not sure I can say exactly what seems lighthearted about the email to me. Perhaps I just assumed it would be, and so read it that way. If I were to pick a few concrete things, I would say the phrase “with our honor intact” seems like a joke, and also “the opportunity to not destroy LessWrong” seems like a silly phrase (kind of similar to “a little bit the worst thing ever”). On reflection, yep, you are getting an opportunity you don’t normally get. But it’s also weird to have an opportunity to perform a negative action.
Also, it still seems to me that there’s no reason anyone who was taking it seriously would blow up LW (apart from maybe Jeff Kauffman). So if there’s a real risk of someone blowing it up, it must not be that serious.
To add some missing context to this:
-I’m part of the EA community and have been for several years. To the extent that you need a community member to blame for this, it is me. When doing this, I was operating under the belief that the community would be judging me personally, which is why I openly admitted to doing this on Facebook.
-I would have known about Petrov Day anyway regardless of Chris’ message.
-Phishing attacks can often have in excess of 80% success rate. If you had received this, you would have likely entered the codes as well, even though everyone thinks that they wouldn’t. Which is just one of the reasons why it doesn’t make sense to punish recipients for making this kind of mistake.
-The campaign wasn’t targeted at Chris, it was sent to lots of users. Retrospectively, I should have excluded Chris from the list of users. (I really regret not doing this, and I would like to apologise to Chris for this.)
Source please on the 80% success rate of many phishing attacks? This is at least an order of magnitude more than I would have predicted, it blows my mind!
Did a quick google. The only statistic I could find for how successful phishing attacks are is https://www.helpnetsecurity.com/2019/09/04/sme-phishing-attacks/:
66% is still way more than I expect, but there’s no verifiable source. (Looks like CybSafe has incentive to exaggerate the numbers.) And it’s not clear whether this is “66% of phishing attempts were successful” or “of organizations targeted, 66% suffered at least one successful attack”. Certainly it doesn’t support “you would have likely entered the codes as well”.
Strong-downvoted pdaa’s comment pending source.
Yeah. If 80% is the true success rate I would not expect the world to look the way it does. I would expect such attacks to be incredibly rampant and somewhere near the front of everyone’s minds, at least in the sense of when you get a call from an unknown number you think it’s quite likely that it’s something unsolicited.
Seconding Daniel’s request for a source. But also, to clarify, does your attempt here count as one phishing attack in total, or one per message you sent?
If it’s one per message, then 80% is double-plus-super-higher-than-predicted. But if it’s one in total, then “you would have likely entered the codes as well” needs further justification. I said in the other thread that I wasn’t super confident I wouldn’t have fallen for it; but I don’t think it’s actively likely that I would have done, even taking your claim into account.
As a note, to the extent that you’re trying to actively shoulder the blame here (rather than simply describing where you think it falls), this isn’t a call you get to make. I’m not saying here that Chris does deserve blame; just that to the extent he does, you can’t take that away from him onto yourself.
And… having this expectation seems like kind of the same sort of thing that went wrong with the admins’ messaging? Like, on a high level you could describe what led to the site blowing up as: “the admins expected people to feel one way about a thing, and acted on that expectation, but some people felt a different way, and acted in ways that surprised the admins”. Similarly, you may have expected us to feel one way about your actions, such that we judge you personally; but if some of us feel a different way, and judge differently, well...
You said you wanted this to be a learning opportunity for the community, and I think (despite varying levels of annoyance) we’re overall taking it as such. To the extent that it’s a learning opportunity for you as well, I hope you take it as such.
“The campaign wasn’t targeted at Chris, it was sent to lots of users. Retrospectively, I should have excluded Chris from the list of users. (I really regret not doing this, and I would like to apologise to Chris for this.)”—I don’t know why you wouldn’t consider me fair game. You really don’t need to apologise to me.
Was the parallel to the real Petrov’s false alarm intentional or unintentional?
I think it’s great that you did this. It made the game more real, and hopefully the rest of us learned something.
Suppose phishing attacks do have an 80%+ success rate. I have been the target of phishing attempts 10s of times, and never fallen for it (and I imagine this is not unusual on LW). This suggests the average LWer should not expect to fall victim to a phishing attempt with 80% probability even if that is the global average
There seem to be multiple meta- games
press the button or not
take it as a game | as a serious ritual | as a serious experiment
cooperate or defect on the implicit rule allowing play behaviour ~ “you are allowed to play and experiment in games and this is safe. it is understood actions you take within the game will not be used as an evidence of intent outside of the game”. (imagine I play a game of chess with someone and interpret my opponent taking my pieces as literarily trying to harm me)
the meta-game of making the game interesting; cf munchkin
the meta-game of making the experiment valuable for learning
coordination about which of these games we are playing
To me Chris’s story hears like [ press | game | cooperate | ? | ? | ]
Overall while I think there is a lot of value in having a community of people who do not press big red buttons, I also see a lot of value in noticing these other games and “cooperating” in them.
This line gave me an important insight into how you were thinking.
The creators were thinking of it as a community trust-building exercise. But you thought that it was intended to be a role-playing game. So, for you, “cooperate” meant “make the game interesting and entertaining for everyone.” That paints the risk of taking the site down in a very different light.
But the “role-playing game” glasses that you were wearing would have (understandably) made such a statement look like “flavor text”.
Agreed. I was also provided with the codes, and to provide another data point, this is how I thought about it.
The terminal goal isn’t to keep the site up. The question I (immediately) asked myself is whether entering the codes would make it more likely or less likely that people take xrisk seriously in the real world (roughly). I considered this briefly, but I realized that I too was confused about the point of the experiment, and thus decided to leave it alone.
Umm. Grudgingly upvoted.
(For real though, respect for taking the time to write an after-action report of your thinking.)
Serious question—will there be any consequences for your friendship, you think?
Why would there be? I’m sure they saw it as just a game too and it would be extremely hypocritical for me to be annoyed at anyone for that.
Thanks, I’m glad to hear that. :) Also, very thankful that the LW community took this really well.
Beyond that, as for my motivations, aside from curiosity as to whether it would work, etc. I considered that it would be an interesting learning opportunity for the community as well. With actual nukes, random untrusted people also have a part to play. Selecting a small group of people tasked with trying to bring down the site might even be a good addition to future instances of Petrov Day.
For what it’s worth, I took care to ensure that the damage from taking the site down would not be too great. The site was archived elsewhere, and the admins themselves accepted the risk of the site going down by starting this game. If this could have hurt people, I wouldn’t have done it.
Beyond that, loyalty and trust are also very important to me. If the admins had trusted me with the launch codes, I wouldn’t have nuked the site (intentionally).
After thinking more about this experiment, it has got me thinking about the payoff matrices. Is there anyone that would have pressed the button if there was guaranteed anonymity, and thus no personal cost? If so, make a second account—I’d be curious to hear your reasoning. Also, in this case there is no tangible benefit that anyone could get by nuking the site. How do we adapt this to situations where there is a benefit that can be gained by pressing the button?
P.S.: My offer still holds! Admins, if you’re feeling adventurous, give me the codes next year and I’ll prove that I won’t use them!
If I understand you correctly, that won’t work. The identity of the button-presser is not determined by which account pressed the button. It’s determined by the launch code string itself—everyone got a personalised launch code. (Which means that if someone stole and used your personalised code, you’d also get blamed—but that seems fair.)
I read that as “make a second account to say anonymously why you would have done it”.
In the spirit of learning from this, I’d be interested to know how many people you sent the message to and how you chose them etc.
I particularly liked the “You will be asked to complete a short survey afterward” touch—what made you think to include it?
I think maybe 6-8, not sure. I was going to go further but the site went down too quickly. Users were selected based on having a large number of posts.
I wanted something to make it sound realistic. And rationalist/EA culture loves surveys and collecting data. :)
Yeah, that honestly made it feel so real.
Part of me wants to say you plonker for falling for it (as you said, there were plenty of clues, plus the fact that the launch codes weren’t repeated in the second message) but another part of me remembers that I fell victim to a Trojan once so I have some sympathy for you.
Different social norms, I suppose.
I’m trying to think if we ever prank each other or socially engineer each other in my social circle, and the answer is yes but it’s always by doing something really cool — like, an ambiguous package shows up but there’s a thoughtful gift inside.
(Not necessarily expensive — a friend found a textbook on Soviet accounting for me, I got him a hardcover copy of Junichi Saga’s Memories of Silk and Straw. Getting each other nice tea, coffee, soap, sometimes putting it in a funny box so it doesn’t look like what it is. Stuff like that. Sometimes nicer stuff, but it’s not about the money.)
Then I’m trying to think how my circle in general would respond to no-permission-given out-of-scope pranking of someone’s real life community that they’re member of — and yeah, there’d be pretty severe consequences in my social circle if someone did that. If I heard someone did what your buddy did who was currently a friend or acquaintance, they’d be marked as someone incredibly discourteous and much less trustworthy. It would just get marked as… pointless rude destructive behavior.
And it’s pretty tech heavy btw, we do joke around a lot, it’s just when we do pranks it’s almost always at the end a gift or something uplifting.
I don’t mean this to be blunt btw, I just re-read it before posting and it reads more blunt than I meant it to — I was just running through whether this would happen in my social circle, I ran it out mentally, and this is what I came up with.
Obviously, everyone’s different. And that’s of course one of the reasons it’s hard for people to get along. Some sort of meta-lesson, I suppose.
I think this case is fairly different to what you describe. The community organised for this to potentially happen and Chris publicised this fact to his friends. The community decided that it was worth the risk so the damage could be assumed not to be large and having the frontpage going down for 24 hours really isn’t a huge deal.
The actual damage is realisitically the fact that the experiment (and associated metaphor) didn’t work but I feel like the lessons learnt should more than make up for that.
You’re being very kind in far-mode consequentialism here, but come on now.
Making your friend look foolish in front of thousands of people is bad etiquette in most social circles.
I’d kinda assumed that one wouldn’t do this unless they were confident their friend would be ok with it, as indeed seems to be the case.
I am an outsider/lurker, so maybe I just don’t get it, but it seems to me that even if the messaging around this event is changed to make it more clearly serious rather than there being a possible interpretation of all in fun no particular outcome is better than any other, there is a very real (not symbolic) mixed message going on with the way things are currently set up. The first message is hey, we’re doing this really cool ritual and you are invited to participate. The second message is we don’t want our website to go down so don’t do anything (please don’t participate).
This is completely outside the question of what the event actually symbolizes and so on. It’s about, well, as far as I understand the simulacra model, it’s about the simulacra level 1 aspects of the situation.
Someone who didn’t check their email/messages would participate as desired.
There is no clear way to accept the invitation to participate in an active way without threatening the desired outcome.
Maybe this is the point. Maybe it’s meant to be confusing. Maybe the people running the event are deliberately setting up a situation where they are sending mixed messages on purpose, because the event is about mixed/ambiguous messages?
But if mixed messages are not a vital part of the event, then I wonder if it would be a more effective ritual for people who receive codes if there was some way they could publicly revoke their codes. That way they could actively participate in not destroying something, instead of the only way to participate in the desired outcome being to not do anything (to imitate the person who didn’t check their messages).
Yeah, I think this is a real problem. I do think there is something a bit interesting about the ambiguity “The only way to win is not to play”, but there are also a bunch of costs associated with the weird ambiguity, and I am not sure how things weigh up.
The current thing and framing is the best we could come up with in a few dozen hours of collective thinking, but we can probably do better, and maybe the weird ambiguity is too costly, though overall I think we got the basic idea across to most people, despite the ambiguity, so I am hopeful that we can iterate on that and get us all the way everyone getting it.
Haha, I take this sentence like 90%-100% seriously. Don’t you think you lost a little honor in the process? Wouldn’t it be something if a decade down the line LessWrong could say “we’ve played this game for X years, including at least Y users every time and Z users total, and no one has ever entered their launch codes?
But alas...
“Don’t you think you lost a little honor in the process?”—Of course, but some people take this way too seriously.
Seriously: like lionhearted said, thanks for the postmortem! The thought process is important. Even if it meant some hurt feelings and a bit of inconvenience, we still got to learn something here. After all, learning something was the point, right? The more data we gather, the more likely we’ll be better off in similar situations in other contexts.
I’m still processing my thoughts and feelings about this year’s Petrov Day, and what you did. It’s very late here, so I’ll write something tomorrow.
I think we should focus a little bit more on the behaviour of the other participants in this game. Coordinating in order not to have a catastrophic event happening is difficult and takes effort. And just hoping that nobody does anything foolish seems to be a strategy doomed to fail in the long run.
Therefore those other participants who took this experiment really seriously might have done much more to prevent this outcome. E.g. forming a small group, announcing that they are dedicated to the front page not being nuked and that everybody seriously thinking about pressing the button should talk about that first. If across the time zones such a group had formed then Chris might have been convinced not to do it.
I don’t mean to belabor the point, but if this were meant as a deadly serious type of exercise I would have expected a much harsher penalty for submitting the launch codes. Why take just the home page down and not the rest of the site? Why only 24 hours? If it were a deadly serious type of exercise, I’d expect in the ballpark of taking down the whole site for a week to a month. I can also see taking it down for a year. Doing so would really hammer home how important xrisk is, which I think would be a very positive outcome and thus a potentially reasonable thing to do.
I have read about one possible case of false nuclear alarm which involveв something like fishing attack. Not sure if it was real or not, and I can’t find the story now, but it could be real or could be creepypasta. Below is what I remember:
In 50s, nuclear-tipped US cruise missiles were stationed in Okinawa in several locations. One location got an obviously false (for some reasons) lunch command: the procedure was incorrect. They recognised it as false and decided to wait for clarification. But another location nearby recognised the command as legit and started preparing the launch. They had to send armed personal to stop them from launching the cruise missile, and some kind of standoff happened, but nobody was killed. During this, a clarification arrived that there is no launch order. No information about who send the false command were ever provided and everybody signed NDA.
It seems not to be clear if it really happened that way:
Bulletin of the Atomic Scientists
Stars and Stripes
At least you made it an actual game instead of a ritual. Thank you for this!
(Reading Faust is one thing, picking a side is another.)
I like to imagine a future LessWrong with 1 million users, and not one pressing the button. That would be very inspiring and a strong signal of being a high trust culture.
I’m pretty sympathetic to this. I’m a LessWrong admin, last year on Petrov day, and someone had talked about selling codes, I considered my price. $10,000 is a meaningful some to me and I think was my thought. I don’t remember what his final number was, but I recall Habryka stating a higher number and me than getting an explanation of why this exercise was much more important than I initially thought (more than $10k). If that explanation of its importance was written somewhere, I don’t recall where.
So yeah, kinda fair on that front. I should probably apologize for giving you quite as hard time about it – just because I’ve internalized the importance.
Wow. I honestly don’t get it—do you have a link to the previous discussion that justified why anyone’s taking it all that seriously?
IMO, this was a completely optional, artificial setup—“just a game”, in Chris’s words. When I got the e-mail, I wondered if it was already down, and was surprised that it wasn’t (though maybe I just didn’t notice—it never seemed down to me, but I go straight to /allPosts without ever looking at the front page).
There was none of the weight of Petrov’s decision, and no tension about picking one or the other—no lasting harm for pressing the button, no violation of norms (or being executed for treason, or losing WWIII) by failing to do so if it were necessary. And no evidence one way or the other what the actual territory is. Really, just a game. And not even a very good one.
The fundamental cooperation to take down the site had ALREADY HAPPENED. When someone wrote the code that would do so if someone pressed the button, that’s FAR FAR stronger than some rando actually pressing the button.
Here was my analysis last year —
https://www.lesswrong.com/posts/vvzfFcbmKgEsDBRHh/honoring-petrov-day-on-lesswrong-in-2019?commentId=ZZ87dbYiGDu6uMtF8
In fairness, my values diverge pretty substantially from a lot of the community here, particularly around “life is serious” vs “life isn’t very serious” and the value of abstract bonds/ties/loyalties/camaraderie.
Thanks. I am not convinced, but I have a better idea of where our perspectives differ. I have to admit this feels a bit like a relationship shit-test, where an artificial situation is created, and far too much weight is put on the result.
I’d be interested to hear various participants’ and observers’ takes on the actual impact of this event, in terms of what they believe about people’s willingness to support the site or improve the world in non-artificial conditions.
Hmm. Appreciate your reply. I think there’s a subtle difference here, let me think about it some.
Hmm.
Okay.
Thrashing it out a bit more, I do think a lot of semi-artificial situations are predictive of future behavior.
Actually, to use an obviously extreme example that doesn’t universally apply, that’s more-or-less the theory behind the various Special Forces selection procedures —
https://bootcampmilitaryfitnessinstitute.com/media/tv-documentaries/elite-special-forces-documentaries/
As opposed to someone artificially creating a conflict to see how the other party navigates it — which I’m not at all a fan of — I think exercises in shared trust have both predictive value for future behavior and build good team cohesion when overcome.
Me too, but I’d ideally want the data captured semi-anonymously. Most people, especially effective people, won’t comment publicly “I think this is despicable and have incremented downwards various confidences in people as a result” whereas the “aww it’s ok, no big deal” position is much more easily vocalized.
(Personally, I’m trying to tone down that type of vocalization myself. It’s unproductive on an individual level — it makes people dislike you for minimal gain. But I speculate that the absence of that level of dialogue and expression of genuine sentiment potentially leads to evaporative cooling of people who believe in teamwork, mission, mutual trust, etc.)
Reasonable minds can differ on this and related points, of course. And I’m very aware my values diverge a bit from many here, again around stuff like seriousness/camaraderie/cohesion/intensity/harm-vs-care/self-expression/defection/etc.
Thanks for your explanation