I want to point out a few things in particular. Firstly, the email was sent out to 270 users which from my perspective made it seem that the website was almost guaranteed to go down at some time, with the only question being when (I was aware game was played last year, but I had no memory of the outcome or the number of users).
I mean, this is a fine judgement to make, but also a straightforwardly wrong one. Last year we had ~150 people, and the site did not go down, with many people saying that we really have to add more incentives if we want to have any substantial chance of the site going down. I do think it’s a pretty understandable mistake to make, but also one that is actually really important to avoid in real-life unilateralist situations.
Obviously, someone pressing the button wouldn’t damage the honor or reputation of Less Wrong and so it seemed to indicate that this was just a bit of fun..
Of course it damaged our reputation! How could it not have? Being able to coordinate on this is a pretty substantial achievement, and failing on this is a pretty straightforwardly sad thing to happen. I definitely lost a good amount of trust in LessWrong, and I know of at least 10 other people who independently expressed similar things. Again, it’s an understandable mistake to make, but also straightforwardly a retrospectively wrong statement.
Now Habryka is annoyed because he was trying to run a specific experiment and that experiment wasn’t, “Can people who kind of care about the game, but don’t care too much get fooled into taking down the site”. I can understand that, I imagine that this experiment took a lot of time to set up and he was probably looking forward to it for a while.
To be clear, I think in real-life situations, people taking the consequences of their actions not seriously and treating things as just a game to be played is a serious path towards real-life risks! I don’t think you destroyed the setup for this experiment at all. Indeed, someone not thinking for very long about the consequences of their actions, and taking an action with pretty serious consequences out of carelessness is one of the primary ways in which I expected the frontpage to get nuked, and was an intentional part of the test that I wanted to perform. Indeed, usually norms deteriorate by people disassociating from them, and saying that they never felt they were real in the first place, and brushing things off as inconsequential.
To clarify this some more, in a bit of a rambling way: different people have different values. While it is obvious to us in this community that destroying civilization via a nuclear war is pretty bad, there are many people who when asked “if you could wipe out humanity with the press of a button” would happily go and press it, especially if they didn’t think much about it, because they have the cached belief that civilization is probably overall bad and that life would be better off without humanity. Or many people believe in an afterlife and that the apocalypse would overall cause there to be less sin, or whatever.
I assign substantial probability to the world being destroyed not by someone who wants to destroy the world, but by someone who just doesn’t really think that their actions will have substantial consequences. Like Petrov could have just been a normal bureaucrat, doing his job, following the protocols that were set out by him, and it’s really not hard to imagine a Petrov who just didn’t really care about his job. Who realized in the abstract that nuclear war was a thing, but didn’t really care about it viscerally, and when given the read from the instruments, just didn’t think about it very hard and forwarded the signals to his superiors. That’s how bad things happen. The most likely world in which civilization ended because Petrov didn’t intervene, seems to me to be one where Petrov’s attitude was overall pretty similar to your attitude here. That doesn’t mean your attitude is wrong, I believe that you would actually care about the real case of the nuclear weapons, and am not at all saying that you specifically wouldn’t have done the right thing if the real deal was on the line, but that the reference class of the reason why you didn’t do the right thing here (by my lights) is pretty representative of the reference class by which I expect things to go wrong in reality.
Communicating and coordinating on shared priorities and values is really hard. It’s a way lots of things break. In this case, we clearly failed at that. But also, that’s part of the challenge of building a real and important thing. In real-life, you don’t get to assume that everyone who is working with you actually really cares about avoiding nuclear war with your enemies. You don’t get to assume that everyone has a shared understanding of humanity being really important to preserve, and that being cautious with humanity’s future is of utmost importance. Most people don’t viscerally believe those statements, so you can’t just build coordination on that assumption, and if you do it anyways, I think things will fail in pretty analogous ways to how they failed on Saturday.
Of course it damaged our reputation! How could it not have? Being able to coordinate on this is a pretty substantial achievement, and failing on this is a pretty straightforwardly sad thing to happen.
I think this is only true conditional on the Petrov Day LW Button being a Serious Thing. But the whole question is whether or not we should consider it a Serious Thing in the first place. Outsiders likely won’t, they’ll just see it as a game.
More generally, the Schelling choice is rabbit not stag—where Rabbit is “don’t take this seriously” and Stag is “do take this seriously”. By putting this thing online and expecting everyone to take it seriously, but without providing a really solid justification for why they should, you’re choosing Stag without prior coordination, which is generally a bad strategy.
I think this is only true conditional on the Petrov Day LW Button being a Serious Thing. But the whole question is whether or not we should consider it a Serious Thing in the first place.
Hmm, I do think this is right. But I do think the payoff matrix here is pretty asymmetric. Like, I think it’s obvious that on-net, given reasonable levels of ambiguity, you will lose some reputation. There is a question of how much, but I think it’s pretty reasonable to assume that you will lose some good amount, because at least some fraction of people will take it seriously.
Like, I do think that it’s fine to push back and say that it should just be a game, and that the people who are taking it seriously are wrong, but as a statement about social reality, predicting that it will not cost you reputation just seems like a wrong prediction.
Like, I am fine with the statement that it shouldn’t cost you reputation. But saying that it won’t cost you reputation, feels pretty wrong.
I think we lose some reputation if people think that we are unable to choose Stag even in Serious Situations. But the main thing that signals to outsiders that this is a Serious Situation instead of a fun game is the disappointed reactions after someone chooses Rabbit. (By default outsiders are much less likely than insiders to think of this sort of thing as serious, and it was already ambiguous enough that many insiders didn’t think of it as serious). If the community reaction was more like “What a great learning experience” and “This is a super interesting outcome” then I doubt there’d be a significant reputational cost. I’d estimate that the cost in weirdness points of running this event in the first place is about an order of magnitude higher.
An analogy: suppose the military practises a war game and sometimes fails to achieve its goal. I don’t think this means they lose reputation. In fact, for certain classes of games, you lose more reputation by always succeeding in your goal, because that means that the goals are rigged. Same here: maybe the LW team sent out the invitations in such a way that they were very confident someone would push the button, or maybe in a way where they were very confident nobody would; I can’t tell from the outside.
Yeah, to be clear, I do think it is actually a valuable signal to have failed at the Petrov Day goal at least once, because it signals pretty credibly that things are not rigged, and failure is possible.
I do also think that if you want your war game to be taken seriously as a sign of your competence, it’s important that both you and the people you were war-gaming against were playing seriously. This doesn’t mean that the war-game had to be a “Serious situation”, but it does mean that your soldiers shouldn’t have just gone “lol, it’s just a game” and started playing cards or something because they got bored.
Like, sure, we could make this just a fun game, which would cause us to also not have to be worried about reputational risks, but I don’t see much value in the version of this that is just a fun game, with no serious component. I am not super confident about the right balance of seriousness and fun, but I am pretty confident that a world where nobody took this seriously just doesn’t seem very interesting to me. It doesn’t allow me to build any real trust with anyone else, and feels like it deteroriates the real and important lessons we can learn from Petrov Day.
“Of course it damaged our reputation! How could it not have?”—I suppose it might if it’s a serious exercise and a lot more people seem to interpet it that way than I expected
I mean, this is a fine judgement to make, but also a straightforwardly wrong one. Last year we had ~150 people, and the site did not go down, with many people saying that we really have to add more incentives if we want to have any substantial chance of the site going down. I do think it’s a pretty understandable mistake to make, but also one that is actually really important to avoid in real-life unilateralist situations.
Of course it damaged our reputation! How could it not have? Being able to coordinate on this is a pretty substantial achievement, and failing on this is a pretty straightforwardly sad thing to happen. I definitely lost a good amount of trust in LessWrong, and I know of at least 10 other people who independently expressed similar things. Again, it’s an understandable mistake to make, but also straightforwardly a retrospectively wrong statement.
To be clear, I think in real-life situations, people taking the consequences of their actions not seriously and treating things as just a game to be played is a serious path towards real-life risks! I don’t think you destroyed the setup for this experiment at all. Indeed, someone not thinking for very long about the consequences of their actions, and taking an action with pretty serious consequences out of carelessness is one of the primary ways in which I expected the frontpage to get nuked, and was an intentional part of the test that I wanted to perform. Indeed, usually norms deteriorate by people disassociating from them, and saying that they never felt they were real in the first place, and brushing things off as inconsequential.
To clarify this some more, in a bit of a rambling way: different people have different values. While it is obvious to us in this community that destroying civilization via a nuclear war is pretty bad, there are many people who when asked “if you could wipe out humanity with the press of a button” would happily go and press it, especially if they didn’t think much about it, because they have the cached belief that civilization is probably overall bad and that life would be better off without humanity. Or many people believe in an afterlife and that the apocalypse would overall cause there to be less sin, or whatever.
I assign substantial probability to the world being destroyed not by someone who wants to destroy the world, but by someone who just doesn’t really think that their actions will have substantial consequences. Like Petrov could have just been a normal bureaucrat, doing his job, following the protocols that were set out by him, and it’s really not hard to imagine a Petrov who just didn’t really care about his job. Who realized in the abstract that nuclear war was a thing, but didn’t really care about it viscerally, and when given the read from the instruments, just didn’t think about it very hard and forwarded the signals to his superiors. That’s how bad things happen. The most likely world in which civilization ended because Petrov didn’t intervene, seems to me to be one where Petrov’s attitude was overall pretty similar to your attitude here. That doesn’t mean your attitude is wrong, I believe that you would actually care about the real case of the nuclear weapons, and am not at all saying that you specifically wouldn’t have done the right thing if the real deal was on the line, but that the reference class of the reason why you didn’t do the right thing here (by my lights) is pretty representative of the reference class by which I expect things to go wrong in reality.
Communicating and coordinating on shared priorities and values is really hard. It’s a way lots of things break. In this case, we clearly failed at that. But also, that’s part of the challenge of building a real and important thing. In real-life, you don’t get to assume that everyone who is working with you actually really cares about avoiding nuclear war with your enemies. You don’t get to assume that everyone has a shared understanding of humanity being really important to preserve, and that being cautious with humanity’s future is of utmost importance. Most people don’t viscerally believe those statements, so you can’t just build coordination on that assumption, and if you do it anyways, I think things will fail in pretty analogous ways to how they failed on Saturday.
I think this is only true conditional on the Petrov Day LW Button being a Serious Thing. But the whole question is whether or not we should consider it a Serious Thing in the first place. Outsiders likely won’t, they’ll just see it as a game.
More generally, the Schelling choice is rabbit not stag—where Rabbit is “don’t take this seriously” and Stag is “do take this seriously”. By putting this thing online and expecting everyone to take it seriously, but without providing a really solid justification for why they should, you’re choosing Stag without prior coordination, which is generally a bad strategy.
(I also endorse Kaj and Neel’s comments below).
Hmm, I do think this is right. But I do think the payoff matrix here is pretty asymmetric. Like, I think it’s obvious that on-net, given reasonable levels of ambiguity, you will lose some reputation. There is a question of how much, but I think it’s pretty reasonable to assume that you will lose some good amount, because at least some fraction of people will take it seriously.
Like, I do think that it’s fine to push back and say that it should just be a game, and that the people who are taking it seriously are wrong, but as a statement about social reality, predicting that it will not cost you reputation just seems like a wrong prediction.
Like, I am fine with the statement that it shouldn’t cost you reputation. But saying that it won’t cost you reputation, feels pretty wrong.
I think we lose some reputation if people think that we are unable to choose Stag even in Serious Situations. But the main thing that signals to outsiders that this is a Serious Situation instead of a fun game is the disappointed reactions after someone chooses Rabbit. (By default outsiders are much less likely than insiders to think of this sort of thing as serious, and it was already ambiguous enough that many insiders didn’t think of it as serious). If the community reaction was more like “What a great learning experience” and “This is a super interesting outcome” then I doubt there’d be a significant reputational cost. I’d estimate that the cost in weirdness points of running this event in the first place is about an order of magnitude higher.
An analogy: suppose the military practises a war game and sometimes fails to achieve its goal. I don’t think this means they lose reputation. In fact, for certain classes of games, you lose more reputation by always succeeding in your goal, because that means that the goals are rigged. Same here: maybe the LW team sent out the invitations in such a way that they were very confident someone would push the button, or maybe in a way where they were very confident nobody would; I can’t tell from the outside.
Yeah, to be clear, I do think it is actually a valuable signal to have failed at the Petrov Day goal at least once, because it signals pretty credibly that things are not rigged, and failure is possible.
I do also think that if you want your war game to be taken seriously as a sign of your competence, it’s important that both you and the people you were war-gaming against were playing seriously. This doesn’t mean that the war-game had to be a “Serious situation”, but it does mean that your soldiers shouldn’t have just gone “lol, it’s just a game” and started playing cards or something because they got bored.
Like, sure, we could make this just a fun game, which would cause us to also not have to be worried about reputational risks, but I don’t see much value in the version of this that is just a fun game, with no serious component. I am not super confident about the right balance of seriousness and fun, but I am pretty confident that a world where nobody took this seriously just doesn’t seem very interesting to me. It doesn’t allow me to build any real trust with anyone else, and feels like it deteroriates the real and important lessons we can learn from Petrov Day.
“Of course it damaged our reputation! How could it not have?”—I suppose it might if it’s a serious exercise and a lot more people seem to interpet it that way than I expected