Team Lead for LessWrong
Ruby
Karma: 12,005
I see interesting points on both sides here. Something about how this comment(s) is expressed makes me feel uncomfortable, like this isn’t the right tone for exploring disagreements about correct moral/cooperative behavior, it at least it makes it a lot harder for me to participate. I think it’s something like it feels like performing moral outrage/indignation in a way that feels more persuadey than explainy, and more in the direction of social pressure, norms-enforcery. The phrase “shame on you” is a particularly clear thing I’ll point at that makes me perceive this.
I was going to write stuff about integrity, and there’s stuff to that, but the thing that is striking me most right now is that the whole effort seemed very incompetent and naive. And that’s upsetting.
I am now feeling uncertain about the incompetence and naivety of it. Whether this was the best move possible that failed to work out, or best move possible that actually did get a good outcome, or a total blunder is determined by info I don’t have.
I have some feeling of they were playing against a higher-level political player which both makes it hard but also means they needed to account for that? Their own level might be 80+th percentile in reference class of executive/board type-people, but still lower than Sam.
The piece that does seem most like they really made a mistake was trying to appoint an interim CEO (Mira) who didn’t want the role. It seems like before doing that, you should be confident the person wants it.
I’ve seen it raised that the board might find the outcome to be positive (board stays independent even if current members leave?). If that’s true, does change the evaluation of the competence. Feels hard for me to confidently judge that, though my gut sense is Sam got more of what he wanted/common knowledge of his sway than others.
Styling of the headers in this post is off and makes it harder to read. Maybe the result of a bad copy/paste?
These recent events have me thinking the opposite: policy and cooperation approaches to making AI go well are doomed – while many people are starting to take AI risk seriously, not enough are, and those who are worried will fail to restrain those who aren’t (where not being risked in a consequence of humans often being quite insane when incentives are at play). The hope lies in somehow developing enough useful AI theory that leading labs adopt and resultantly build an aligned AI even though they never believed they were going to cause AGI ruin.
And so maybe let’s just get everyone to focus on the technical stuff. Actually more doable than wrangling other people to not build unsafe stuff.
If he can lead an exodus from OpenAI to Microsoft, he can lead one from Microsoft to somewhere else.
People associated with EA are likely to decide at some point that the normal rules for the organization do not apply to them, if they expect that they can generate a large enough positive impact in the world by disregarding those rules.
I am myself consequentialist at my core, but invoking consequentialism to justify breaking commitments, non-cooperation, theft, or whatever else is just a stupid bad policy (the notion of people doing this generates some strong emotions for me) that as a policy/algorithm, won’t result in accomplishing one’s consequentialist goals.
I fear what you say is not wholly inaccurate and is true of at least some in EA, I hope though it’s not that true of many.
Where it does get tricky is potentially unilateral pivotal acts about which I think go in this direction but also feel different from what you describe.
Dialogue on the Claim: “OpenAI’s Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI”
Is the Wave non-disparagement thingy okay?
You were the first, as you guessed.
If I were to do it again, I might include such an option, though I’m still not terribly sad I didn’t.
If we really wanted that info, I could sample and survey people who received the message, looked at it (we have data to know this) and ask them why they didn’t vote. My guess is between 1-10% of people who didn’t vote because of frame commented about it,, so that’s 40 to 400.
372 people have responded by now out of 2500, so 15%. Let’s guess that 50% of people saw it by now, so ~1250 (though could get better data on it). A third responded if so, which seems great for a poll. Of those the 800 who saw but didn’t, I could see 100-400 doing so because the frame didn’t really seem right (lining up with the above estimate). Which seems fine. I bet if I’d spent 10x developing the poll, I wouldn’t get that number down much, and knowing it with more precision doesn’t really help. It’s LW, people are very picky and precise (a virtue...but also makes having some nice things hard).
Sorry, I get the point that the option provided doesn’t let you mu/reject the frame. It’s not clear to me that this is a core framing of Petrov’s actions/virtue was conscientious objection or so on.
Beyond that, the survey wasn’t aiming to allow people to symbolically act out their responses or to reject the frame in an unambiguous way. Insisting that you get to register that you saw it but didn’t like it feels like insisting that you get to participate, but in your own way, rather than simply not engaging if you don’t like it. I also feel like if there was an option to conscientious object and you took that, that’d still be within the frame I created for you to do so? But open to being corrected here.
Huh, I’m surprised that happened. I wouldn’t have thought you’d get a message given that.
I didn’t list in the main post or say until because I fear that I’m saying it defensively in response to criticism, but to model the design for this year requires knowing that we spent vastly less time on it, deciding to something at the last minute. (We’d been very busy with a massive conference in days before Petrov Day.
At 11am (US West Coast time) we started thinking there was something we could maybe do, and at 12pm we got started. I felt we needed to rush if we were to include European folks at all, so really was looking for something we could get done quickly. As the post mentions, we didn’t spend much time on the poll options or trying to design it well, we just wanted something out so half the people wouldn’t be completely excluded.
The second message idea wasn’t even chosen until about half an hour before I sent it. We basically sent the first message and then worked on figuring out how to build on it, and then “next year’s will be decided based on this” was an 11th hour insight. It gives it some stakes without being overwhelming stakes.
Could we have done something better with more time and effort? For sure. But I think this was better than letting Petrov Day pass without any kind of commemoration.
I predict that, if you presented the parts I quoted from the survey message to a random sample of the university-educated population, and asked them whether they thought the poll was biased, >50% would say yes
That seems quite plausible to me. My response was that we weren’t trying especially hard to avoid bias because we weren’t trying to get a super clear result.
And as such, and in keeping with the Petrov Day theme, I maintain that it’s important to offer a true “mu” option, or a “do not participate”.
Can you elaborate on this?
My metahonesty is I might hoodwink you a little on April Fools, Petrov Day, and similar.
I’m sad you didn’t like it. It indeed was not a carefully planned rigorous survey of Petrov Day attitudes.
In my thinking, it was more the start to a ~game/exercise than trying to maximally model people’s attitudes. I wanted to assign people to “teams” (I’d considered random assignment), but this felt this was a little more meaningful, and there’s non-zero bits even in an imperfect survey.
There was no intention to be leading in the responses, nor to corral for any particular response.
I actually hoped that the slap-dash nature would make people suspicious (plus inadvertent bugs/typos) and get them more into Petrov Day mood. From other comments, it sounds like this did happen somewhat.
I think if a failure happened here, it’s that you and others saw the poll as primarily an attempt to accurate survey LessWrong member’s beliefs about Petrov (pretty reasonable belief), but for me it was the start to something else, and goal wasn’t “rigorous survey”, for which a mu option would have made sense.
I’m uncertain how much we should ever be a little sneaky/misleading for the sake of games/experiments/etc. I’m a pro norm that on April Fool’s and Petrov Day and similar, people might hoodwink you a little. At least I might, I will as say a matter of metahonesty.
There is an upside to being the kind of person who will press the button in retaliation. You hope never to, but the fact that you credibly would allows for MAD game theory to apply. (FDT, etc. etc.)
Thanks for sharing all of that in such detail, <3 You make me feel quite glad we did this celebration.
Would you like to know which number click yours was?
Oh, good catch. I had the rows on the denominator sorted wrong so that table was 75% wrong. Fixed now...
My intuition (not rigorous) is there a multiple levels in the consequentialist/deontoligical/consequentialist dealio.
I believe that unconditional friendship is approximately something one can enter into, but one enters into it for contingent reasons (perhaps in a Newcomb-like way – I’ll unconditionally be your friend because I’m betting that you’ll unconditionally be my friend). Your ability to credibly enter such relationships (at least in my conception of them) is dependent on you not starting to be more “conditional” because you doubt that the other person is also being there. This I think is related to not being a “fair weather” friend. I continue to be your friend even when it’s not fun (you’re sick, need taking care of whatever) even if I wouldn’t have become your friend to do that. And vice versa. Kind of a mutual insurance policy.
Same thing could be with contracts, agreements, and other collaborations. In a Newcomb-like way, I commit to being honest, being cooperative, etc to a very high degree even in the face of doubts about you. (Maybe you stop by the time someone is threatening your family, not sure what Ben, et al, think about that.) But the fact I entered into this commitment was based on the probabilities I assigned to your behavior at the start.