orthonormal

Karma: 16,889

orthonormal 22 Nov 2023 20:38 UTC
10 points
2
in reply to: SkinnyTy’s comment on: OpenAI: The Battle of the Board
Prior to hiring Shear, the board offered a merger to Dario Amodei, with Dario to lead the merged entity. Dario rejected the offer.

orthonormal 22 Nov 2023 0:33 UTC
8 points
3
on: Dialogue on the Claim: “OpenAI’s Firing of Sam Altman (And Shortly-Subsequent Events) On Net Reduced Existential Risk From AGI”
I mean, I don’t really care how much e.g. Facebook AI thinks they’re racing right now. They’re not in the game at this point.
The race dynamics are not just about who’s leading. FB is 1-2 years behind (looking at LLM metrics), and it doesn’t seem like they’re getting further behind OpenAI/Anthropic with each generation, so I expect that the lag at the end will be at most a few years.
That means that if Facebook is unconstrained, the leading labs have only that much time to slow down for safety (or prepare a pivotal act) as they approach AGI before Facebook gets there with total recklessness.
If Microsoft!OpenAI lags the new leaders by less than FB (and I think that’s likely to be the case), that shortens the safety window further.
I suspect my actual crux with you is your belief (correct me if I’m misinterpreting you) that your research program will solve alignment and that it will not take much of a safety window for the leading lab to incorporate the solution, and therefore the only thing that matters is finishing the solution and getting the leading lab on board. It would be very nice if you were right, but I put a low probability on it.

orthonormal 21 Nov 2023 20:18 UTC
40 points
7
on: OpenAI: Facts from a Weekend
I’m surprised that nobody has yet brought up the development that the board offered Dario Amodei the position as a merger with Anthropic (and Dario said no!).
(There’s no additional important content in the original article by The Information, so I linked the Reuters paywall-free version.)
Crucially, this doesn’t tell us in what order the board made this offer to Dario and the other known figures (GitHub CEO Nat Friedman and Scale AI CEO Alex Wang) before getting Emmett Shear, but it’s plausible that merging with Anthropic was Plan A all along. Moreover, I strongly suspect that the bad blood between Sam and the Anthropic team was strong enough that Sam had to be ousted in order for a merger to be possible.
So under this hypothesis, the board decided it was important to merge with Anthropic (probably to slow the arms race), booted Sam (using the additional fig leaf of whatever lies he’s been caught in), immediately asked Dario and were surprised when he rejected them, did not have an adequate backup plan, and have been scrambling ever since.
P.S. Shear is known to be very much on record worrying that alignment is necessary and not likely to be easy; I’m curious what Friedman and Wang are on record as saying about AI x-risk.

orthonormal 21 Nov 2023 20:02 UTC
3 points
0
in reply to: Chess3D’s comment on: OpenAI: Facts from a Weekend
No, I don’t think the board’s motives were power politics; I’m saying that they failed to account for the kind of political power moves that Sam would make in response.

orthonormal 20 Nov 2023 20:58 UTC
10 points
2
in reply to: johnswentworth’s comment on: Sam Altman, Greg Brockman and others from OpenAI join Microsoft
In addition to this, Microsoft will exert greater pressure to extract mundane commercial utility from models, compared to pushing forward the frontier. Not sure how much that compensates for the second round of evaporative cooling of the safety-minded.

orthonormal 20 Nov 2023 20:13 UTC
33 points
20
in reply to: Lucius Bushnaq’s comment on: OpenAI: Facts from a Weekend
If they thought this would be the outcome of firing Sam, they would not have done so.
The risk they took was calculated, but man, are they bad at politics.

orthonormal 20 Nov 2023 20:10 UTC
26 points
9
in reply to: ryan_b’s comment on: OpenAI: Facts from a Weekend
1. The quote is from Emmett Shear, not a board member.
2. The board is also following the “don’t say anything literally false” policy by saying practically nothing publicly.
3. Just as I infer from Shear’s qualifier that the firing did have something to do with safety, I infer from the board’s public silence that their reason for the firing isn’t one that would win back the departing OpenAI members (or would only do so at a cost that’s not worth paying).
4. This is consistent with it being a safety concern shared by the superalignment team (who by and large didn’t sign the statement at first) but not by the rest of OpenAI (who view pushing capabilities forward as a good thing, because like Sam they believe the EV of OpenAI building AGI is better than the EV of unilaterally stopping). That’s my current main hypothesis.

orthonormal 20 Nov 2023 18:27 UTC
14 points
3
in reply to: Lukas_Gloor’s comment on: OpenAI: Facts from a Weekend
It’s too late for a conditional surrender now that Microsoft is a credible threat to get 100% of OpenAI’s capabilities team; Ilya and Jan are communicating unconditional surrender because the alternative is even worse.

orthonormal 20 Nov 2023 18:09 UTC
15 points
1
in reply to: David Hornbein’s comment on: OpenAI: Facts from a Weekend
I agree, it’s critical to have a very close reading of “The board did *not* remove Sam over any specific disagreement on safety”.
This is the kind of situation where every qualifier in a statement needs to be understood as essential—if the statement were true without the word “specific”, then I can’t imagine why that word would have been inserted.

orthonormal 20 Nov 2023 17:22 UTC
61 points
43
on: OpenAI: Facts from a Weekend
The most likely explanation I can think of, for what look like about-faces by Ilya and Jan this morning, is realizing that the worst plausible outcome is exactly what we’re seeing: Sam running a new OpenAI at Microsoft, free of that pesky charter. Any amount of backpedaling, and even resigning in favor of a less safety-conscious board, is preferable to that.
They came at the king and missed.

orthonormal 11 Nov 2023 20:08 UTC
13 points
11
in reply to: Richard_Ngo’s comment on: Integrity in AI Governance and Advocacy
Did anyone at OpenAI explicitly say that a factor in their release cadence was getting the public to wake up about the pace of AI research and start demanding regulation? Because this seems more like a post hoc rationalization for the release policy than like an actual intended outcome.

orthonormal 10 Nov 2023 0:35 UTC
9 points
−1
in reply to: rvnnt’s comment on: Vote on Interesting Disagreements
I expect AGI to emerge as part of the frontier model training run (and thus get a godshatter of human values), rather than only emerging after fine-tuning by a troll (and get a godshatter of reversed values), so I think “humans modified to be happy with something much cheaper than our CEV” is a more likely endstate than “humans suffering” (though, again, both much less likely than “humans dead”).

orthonormal 8 Nov 2023 19:17 UTC
5 points
1
in reply to: rvnnt’s comment on: Vote on Interesting Disagreements
Steelmanning a position I don’t quite hold: non-extinction AI x-risk scenarios aren’t limited to inescapable dystopias as we imagine them.
“Kill all humans” is certainly an instrumental subgoal of “take control of the future lightcone” and it certainly gains an extra epsilon of resources compared to any form of not literally killing all humans, but it’s not literally required, and there are all sorts of weird things the AGI could prefer to do with humanity instead depending on what kind of godshatter it winds up with, most of which are so far outside the realm of human reckoning that I’m not sure it’s reasonable to call them dystopian. (Far outside Weirdtopia, for that matter.)
It still seems very likely to me that a non-aligned superhuman AGI would kill humanity in the process of taking control of the future lightcone, but I’m not as sure of that as I’m sure that it would take control.

orthonormal 8 Nov 2023 18:51 UTC
2 points
0
on: Thinking By The Clock
[See corrections in replies; “think for five minutes” was in EY posts as far back as 2007, the HPMOR chapter was in 2010, and the first CFAR retreat (though not under that name) was 2011 IIRC. Still curious to know where he got it from.]
~~Before HPMOR, “think for five minutes by the clock” was a CFAR exercise; I don’t recall where they picked it up from.~~

orthonormal 28 Oct 2023 17:18 UTC
14 points
8
in reply to: Ben Pace’s comment on: Truthseeking, EA, Simulacra levels, and other stuff
I think a substantial fraction of LWers have the (usually implicit—they may not have even read about simulacra) belief that higher levels are inherently morally problematic, and that engaging on those levels about an important topic is at best excusable under the kind of adversarial circumstances where direct lies are excusable. (There’s the obvious selection effect where people who feel gross about higher levels feel more comfortable on LW than almost anywhere else.)
I think there need to be better public arguments against that viewpoint, not least because I’m not fully convinced it’s wrong.

orthonormal 3 Oct 2023 19:48 UTC
3 points
1
in reply to: Matthew Barnett’s comment on: EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem
Elizabeth has put at least dozens of hours into seeking good RCTs on vegan nutrition, and has come up nearly empty. At this point, if you want to say there is an expert consensus that disagrees with her, you need to find a particular study that you are willing to stand behind, so that we can discuss it. This is why Elizabeth wrote a post on the Adventist study—because that was the best that people were throwing at her.

orthonormal 3 Oct 2023 19:43 UTC
3 points
−1
in reply to: Matthew Barnett’s comment on: EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem
This is a pretty transparent isolated demand for rigor. Can you tell me you’ve never uncritically cited surveys of self-reported data that make veg*n diets look good?

orthonormal 2 Oct 2023 17:37 UTC
2 points
6
in reply to: Noosphere89’s comment on: EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem
Simply type the at-symbol to tag people. I don’t know when LW added this, but I’m glad we have it.

orthonormal 2 Oct 2023 17:34 UTC
2 points
0
in reply to: Iknownothing’s comment on: EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem
Your framing makes it sound like individual raising of livestock, which is silly—specialization of expertise and labor is a very good thing, and “EA reducetarians find or start up a reasonably sized farm whose animal welfare standards seem to them to be net positive” seems to dominate “each EA reducetarian tries to personally raise chickens in a net positive way” (even for those who think both are bad, the second one seems simply worse at a fixed level of consumption).

orthonormal 2 Oct 2023 17:20 UTC
4 points
0
in reply to: Noosphere89’s comment on: EA Vegan Advocacy is not truthseeking, and it’s everyone’s problem
Seems fair to tag @Liron here.