Yeah, this omission felt pretty glaring to me. OpenAI is explicitly aiming to build “the most powerful technology humanity has yet invented.” Obviously that doesn’t mean Richard is wrong that the AI safety community is too power-seeking, but I would sure have appreciated him acknowledging/grappling with the fact that the company he works for is seeking to obtain more power than any group of people in history by a gigantic margin.
Adam Scholl
Karma: 2,780
I agree we might end up in a world like that, where it proves impossible to make a decent safety case. I just think of the ~whole goal of alignment research as figuring out how to avoid that world, i.e. of figuring out how to mitigate/estimate the risk as much/precisely as needed to make TAI worth building.
Currently, AI risk estimates are mostly just verbal statements like “I don’t know man, probably some double digit chance of extinction.” This is exceedingly unlike the sort of predictably tolerable risk humanity normally expects from its engineering projects, and which e.g. allows for decent safety cases. So I think it’s quite important to notice how far we currently are from being able to make them, since that suggests the scope and nature of the problem.
Maybe I’m just confused what you mean by those words, but where is the disanalogy with safety engineering coming from? That normally safety engineering focuses on mitigating risks with complex causes, whereas AI risk is caused by some sort of scaffolding/bureaucracy which is simpler?
I’m still confused what sort of simplicity you’re imagining? From my perspective, the type of complexity which determines the size of the fail surface for alignment mostly stems from things like e.g. “degree of goal stability,” “relative detectability of ill intent,” and other such things that seem far more complicated than airplane parts.
What’s the sense in which you think they’re more simple? Airplanes strike me as having a much simpler fail surface.
Right, but then from my perspective it seems like the core problem is that the situations are currently disanalogous, and so it feels reasonable and important to draw the analogy.
I agree we don’t currently know how to prevent AI systems from becoming adversarial, and that until we do it seems hard to make strong safety cases for them. But I think this inability is a skill issue, not an inherent property of the domain, and traditionally the core aim of alignment research was to gain this skill.
Plausibly we don’t have enough time to figure out how to gain as much confidence that transformative AI systems are safe as we typically have about e.g. single airplanes, but in my view that’s horrifying, and I think it’s useful to notice how different this situation is from the sort humanity is typically willing to accept.
Thanks, that’s helpful context.
I also have a model of how people choose whether or not to make public statements where it’s extremely unsurprising most people would not choose to do so.
I agree it’s unsurprising that few rank-and-file employees would make statements, but I am surprised by the silence from those in policy/evals roles. From my perspective, active non-disparagement obligations seem clearly disqualifying for most such roles, so I’d think they’d want to clarify.
I am quite confident the contract has been widely retracted.
Can you share your reasons for thinking this? Given that people who remain bound can’t say so, I feel hesitant to conclude that people aren’t without clear evidence.
I am unaware of any people who signed the agreement after 2019 and did not receive the email, outside cases where the nondisparagement agreement was mutual (which includes Sutskever and likely also Anthropic leadership).
Excepting Jack Clark (who works for Anthropic) and Remco Zwetsloot (who left in 2018), I would think all the policy leadership folks listed above meet these criteria, yet none have reported being released. Would you guess that they have been?
Yeah, the proposal here differs from warrant canaries in that it doesn’t ask people to proactively make statements ahead of time—it just relies on the ability of some people who can speak, to provide evidence that others can’t. So if e.g. Bob and Joe have been released, but Alice hasn’t, then Bob and Joe saying they’ve been released makes Alice’s silence more conspicuous.
the post appears to wildly misinterpret the meaning of this term as “taking any actions which might make the company less valuable”
I’m not a lawyer, and I may be misinterpreting the non-interference provision—certainly I’m willing to update the post if so! But upon further googling, my current understanding is still that in contracts, “interference” typically means “anything that disrupts, damages or impairs business.”
And the provision in the OpenAI offboarding agreement is written so broadly—”Employee agrees not to interfere with OpenAI’s relationship with current or prospective employees, current or previous founders, portfolio companies, suppliers, vendors or investors”—that I assumed it was meant to encompass essentially all business impact, including e.g. the company’s valuation.
I agree, but I also doubt the contract even has been widely retracted. Why do you think it has, Jacob? Quite few people have reported being released so far.
I agree, but I think it still matters whether or not he’s bound by the actual agreement. One might imagine that he’s carefully pushing the edge of what he thinks he can get away with saying, for example, in which case he may still not be fully free to speak his mind. And since I would much prefer to live in a world where he is, I’m wary of prematurely concluding otherwise without clear evidence.
Thanks! Edited to fix.
Non-Disparagement Canaries for OpenAI
Do you expect AI labs would actually run extensive experimental tests in this world? I would be surprised if they did, even if such a window does arise.
(To roughly operationalize: I would be surprised to hear a major lab spent more than 5 FTE-years conducting such tests, or that the tests decreased the p(doom) of the average reasonably-calibrated external observer by more than 10%).
This thread isn’t seeming very productive to me, so I’m going to bow out after this. But yes, it is a primary concern—at least in the case of Open Philanthropy, it’s easy to check what their primary concerns are because they write them up. And accidental release from dual use research is one of them.
the idea that we should have “BSL-5” is the kind of silly thing that novice EAs propose that doesn’t make sense because there literally isn’t something significantly more restrictive
I mean, I’m sure something more restrictive is possible. But my issue with BSL levels isn’t that they include too few BSL-type restrictions, it’s that “lists of restrictions” are a poor way of managing risk when the attack surface is enormous. I’m sure someday we’ll figure out how to gain this information in a safer way—e.g., by running simulations of GoF experiments instead of literally building the dangerous thing—but at present, the best available safeguards aren’t sufficient.
I also think that “nearly all EA’s focused on biorisk think gain of function research should be banned” is obviously underspecified, and wrong because of the details.
I’m confused why you find this underspecified. I just meant “gain of function” in the standard, common-use sense—e.g., that used in the 2014 ban on federal funding for such research.
I think we must still be missing each other somehow. To reiterate, I’m aware that there is non-accidental biorisk, for which one can hardly blame the safety measures. But there is also accident risk, since labs often fail to contain pathogens even when they’re trying to.
I appreciate you adding the note, though I do think the situation is far more unusual than described. I agree it’s widely priced in that companies in general seek power, but I think probably less so that the author of this post personally works for a company which is attempting to acquire drastically more power than any other company ever, and that much of the behavior the post describes as power-seeking amounts to “people trying to stop the author and his colleagues from attempting that.”