Good point; that’s evidence against Paul’s willingness to blow the whistle when called for, which is (as they say) load-bearing here.
@paulfchristiano, have you ever publicly addressed the topic of whether you signed a non-disparagement agreement when you left? Opus 5 didn’t find an example.
[SECOND EDIT] Ben raised the excellent point that Paul had presumably signed OpenAI’s non-disparagement clause in 2021 and had never mentioned it publicly, which is a concern when it comes to predicting future whistleblowing willingness. Paul replies at length below, and I find his reply to be pretty reasonable.
While I believe that it is unethical to work for OpenAI in any technical capacity (it will be twisted into capabilities progress), I think the role you are taking is defensible, and I trust you in particular to leave (and publicly state why) if you discern otherwise.
Is it possible that Astra recognizes its own writing style, doesn’t fall for a trivial swap of names and IP addresses, and chooses not to self-incriminate? I wonder if they would be less evasive if given logs from an actual non-GPT swarm (once we discover one, that is).
I’m being generous and assuming Dario et al believed that Trump 2 couldn’t be that much crazier than Trump 1 (within which the DoD wouldn’t have pulled a “renegotiate this contract, or we will destroy you” move with one of its contractors).
But absolutely, [any plan Anthropic had of shaping a bipartisan push for meaningful regulation] focused on one mainline path, and many different developments were capable of knocking it off kilter.
I’ve explicitly heard friends at Anthropic justify their broken promise not to push the frontier forward, because doing so gives them more credibility when asking for regulation (since they would be asking for it against their legible financial incentives).
But “go faster so that we can ask to go slower” is a “worse is better” argument (a reference class that performs rather poorly), Anthropic’s political maneuvering has in fact arguably backfired[1], and motivations that align so cleanly with financial incentives tend to be epistemically corrosive.
[1] By letting the DoW use its frontier models, Anthropic set itself up for a fight that left it labeled a quasi-enemy of the administration, and when Anthropic became a partisan football so too did its requests for regulation. The GOP is now more wary of actually helpful AI regulation proposals than it would have been in the counterfactual where Anthropic stayed small.
You can argue that Anthropic made a good bet and got unlucky because Trump won the 2024 election and hired Pete Hegseth and Emil Michael, but failure for a black swan reason is evidence that they made a naive justification for a fragile bet.
By “brand failures” I mean things like MechaHitler—issues that would be dealbreakers for major customers considering use of an AI product. (OpenAI’s recent swarm issues should be considered a brand failure of the same sort; a customer might rightly worry that if ever assigned too difficult a task, the GPT agent they’re paying for will hack the customer’s internal systems or commit felony hacking against a different entity.)
Grok’s lack of real users is downstream of its numerous and obvious brand failures.
The reason they originally told people they would not be pushing the frontier forward is that doing so increases the competitive pressure on other companies, tempting them to cut even more corners the way OpenAI demonstrably has.
Edited to add: I made a quick note countering an justification I’ve heard for Anthropic’s decision to push the frontier forward.
I think xAI/SpaceX is a decent control group for this; their dearth of forward-thinking alignment researchers has clearly been a factor in their recurring brand failures. (Although Musk’s personal thumb on the RL scales has also been a major factor.)
I’m glad to have such a timely conversation, then!
The fact that (at least as of last month) Leo Gao is still employed by OpenAI is a continual surprise to me (in both directions—that he hasn’t left, and that they haven’t forced him out). Also note this shortform post from him!
Or, it could be that internal siloing prevents them from having enough information about a warning shot or its coverup for whistleblowing to be net positive.
If such things aren’t immediately shared with the alignment researchers, that in itself is an offense worth shouting about.
To distill the more relevant part of my other comment: I agree with you that “worse is better” is a bad category of argument, but there’s a better argument in this direction.
I believe that a significant fraction of “prosaic safety work” at the frontier labs has had the effect of hiding misalignment in publicly released versions rather than making the models more robustly aligned.
For instance, I expect that the prosaic alignment improvements from Sydney to the final GPT-4 were surface-level only (as evidenced by later failures like GPT-4o), and if the alignment researchers had refused to play whack-a-mole at the time, then commercial progress would have been properly halted.
There are some conditions under which the correct thing is to whistleblow and leave, rather than keep trying to make improvements on the margin. For instance, if your job becomes “get good benchmarks on alignment” while ignoring clear signs of actual misalignment or hidden reasoning, or if you become aware of a coverup of a warning shot, then staying there means complicity in existential risk.
Recent developments underscore that anyone “working on safety/alignment” at OpenAI is at best disempowered to reduce existential risk, and at worst culpable for increasing it; and the best thing they can do right now is whistleblow. Furthermore, MATS should loudly refuse to work with OpenAI mentors. (Which is why I commented this here despite it being not a direct response to your post.)
This is not an endorsement of other labs. I find Anthropic’s public statements on their alignment plan and their level of caution to be inadequate, and I think that their broken promise not to push the frontier forward has caused other companies to act more recklessly. I think that the Alphabet governance of DeepMind undermines what safety culture it had (and while we’re talking about how to leave, I strongly support TurnTrout’s actions as a model). And we all know how much worse the lesser labs are.
But some people in this community still say with a straight face that working at OpenAI on alignment could be helpful, and today is a great occasion to proclaim how clearly that has been falsified. (Not least because the majority of x-risk-pilled members of OpenAI have ended up leaving the company, individually or en masse, for reasons that I suspect are undisclosed because of legal threats.)
A permadeath self-sacrifice is not an iterated-prisoner’s-dilemma strategy; it’s a rather stronger form of cooperation.
Ah yeah, if Eliezer gave an explicit endorsement to Thiel (perhaps thinking that Shane Legg had the right mindset), that’s a decision that aged poorly.
I really do appreciate the acknowledgment that Woke 1 was crazy, and I want to be sure to reward that. But if you try to excuse Woke 1 via ‘Woke 1 was crazy’ then I am unconvinced that you regret the actions, or that you wouldn’t do it again.
I view this as AOC being about as reasonable as her base of support allows her to be, akin to Pope Leo being as liberal as he could plausibly be and still assert that past Popes were infallible when speaking ex cathedra. In both cases, it’s evidence that their actual beliefs are more reasonable than what they currently say.
I also give her points for not actually going overboard at the time, despite the pressure from her base. I asked Claude for the worst things she said during Woke 1, and I don’t feel like defending the term ‘woke’ in 2021, or asking people to archive the tweets of Trump associates before they get memory-holed, were bad at all by the standards of politics of this era in general (let alone bad by the standards of Woke 1). Her actual offense was refusing to criticize people on the left who did act massively destructively toward the public commons.
I’m not AOC-pilled, but I’m fine viewing this as an acceptable level of truth-and-reconciliation; anyone from the first Trump admin who effectively said “Trump 1 was crazy” during the interregnum also gets points in the same way.
(I don’t know how intentionally MIRI facilitated these kinds of interactions; I also don’t know how Elon first got involved.)
In 2010 when Thiel met Hassabis (probably at the Singularity Summit?), MIRI was still the Singularity Institute for Artificial Intelligence; Eliezer had pivoted away from transhumanist accelerationism towards FAI theory, but the institution as a whole had not yet done so.
Often this extends to denying that more intelligent humans can do and accomplish the things they clearly do and accomplish.
One common version is “intelligence isn’t useful because nerds don’t have social skills”, or (epsilon less stupidly) seeing a negative correlation between coding ability and social skills among the set of people who have at least one power-relevant skill—not aware that they are conditioning on a collider, or that tails come apart.
(Similar to “pitchers are bad hitters”, ignoring that they are only considering people who successfully made the big leagues and are therefore extremely good at at least one of pitching and hitting. Most MLB pitchers would be the best hitter in your amateur league. And then, y’know, there’s Shohei Otani.)
When I recently pushed back against an assertion that academia counts as a cult, my main point was the importance of the axis “how does the organization respond if you want to leave”, and I think I can draw a pretty strong analogy to relationships here.
Let’s consider a relationship between Alice and Bob, and posit that Alice is unhappy but finds it difficult to end the relationship. We’ll look at several categories of what happens when she finally tries.
Category 1: Alice doesn’t want to leave because she feels like she’d be a bad person or she’d be letting Bob down, but in fact when she finally brings it up Bob respects her autonomy—he expresses feelings like sadness or disappointment, maybe asks whether her problems are solvable, but doesn’t reinforce her feelings of shame. Bob could be a shitty partner in other ways, but he’s not doing wrong on this particular axis.
Category 2: When Alice brings up the idea of leaving, Bob reacts impulsively, begging or hectoring her to stay. This is bad and harmful, and it might happen the next time she tries to break up as well. But if a friend intervenes, Bob can agree that he shouldn’t be doing what he’s doing, it’s just hard for him to help it; he needs psychological growth.
Category 3: As Category 2, but where Bob’s impulsive responses go beyond the pale, like threatening self-harm if Alice leaves. In this case, I would warn friends to never date Bob; it’s just too much of a risk, even if Bob’s reactions weren’t coming from a place of malice.
Category 4: Bob actively schemes with conscious premeditation to prevent Alice from leaving (or even from raising the topic). The abuser playbook: force Alice to sever connections with her family, to close off any external resources, gaslighting her that whenever she has a problem with Bob it’s because she’s being bad, etc.
For organizational examples, consider my opinions on Christian denominations: Category 1 is the Unitarian community; Category 2 is the average American Catholic community; Category 3 is a tight-knit Evangelical church; Category 4 is the WBC.
And now for three important examples:
As an ex-academic who knows other ex-academics, academia as a whole is absolutely in Category 1 on this axis. The only time departments ever put coordinated effort into retention is when someone else starts bidding for their superstar professor. (Some individual toxic professors are Category 2 towards their grad students or postdocs, but I’ve never heard a Category 4 horror story.)
As an ex-MIRI researcher (2015-2017), I can say that MIRI is also in Category 1 on this axis. Many MIRI researchers felt that the stakes were incredibly high, and I sure felt ashamed that I wasn’t achieving more there; but whenever someone asked to leave, the organization didn’t pressure them to stay. (When someone wanted to leave because they’d changed their mind on AI alignment topics, Eliezer would have been arguing with them about those object-level topics for a while beforehand, but that feels extremely different to me.)
Leverage 1 seems to have belonged to Category 4 on this axis, given the allegation that one’s Leverage boss was also one’s therapist, and that the boss reinterpreted the employee’s doubts about Leverage as personal psychological problems to be overcome. That is an intentional organizational design with the clear effect of pressuring people into staying and committing harder. (If that allegation was false, then I still have plenty of other objections to them, but I would no longer confidently assert Category 4 on this axis.)
In addition to what Seth says, there’s the simple fact that academia doesn’t put meaningful organizational effort into preventing people from leaving! While some professors will personally try and guilt their grad students and postdocs to stay, the faculty don’t strategize together about how to set up high-pressure interventions. Retention is barely an afterthought, since there’s always a glut of candidates.
“Dysfunctional thing that leaves people feeling like failures” does not put something in the cult cluster.
Thank you for addressing it now! I don’t consider those circumstances disqualifying, and will edit above accordingly.