AI strategy & governance. ailabwatch.org. Looking for new projects.
As of 1 June 2024, I’ve recently focused on blogging, but I expect to soon focus on exploring a version of ailabwatch.org that could get more attention. I’m most excited to receive offers to help with projects like ailabwatch.org. I’m also excited to be pitched blogposts/projects.
Securing model weights is underrated for AI safety. (Even though it’s very highly rated.) If the leading lab can’t stop critical models from leaking to actors that won’t use great deployment safety practices, approximately nothing else matters. Safety techniques would need to be based on properties that those actors are unlikely to reverse (alignment, maybe unlearning) rather than properties that would be undone or that require a particular method of deployment (control techniques, RLHF harmlessness, deployment-time mitigations).
However hard the make a critical model you can safely deploy problem is, the make a critical model that can safely be stolen problem is… much harder.
This post is poorly organized/edited and the important parts aren’t novel; you should go read AI catastrophes and rogue deployments and AI Control and The case for ensuring that powerful AIs are controlled instead.
I mostly agree. Anthropic is kinda an exception (see footnotes 3, 4, 5, 11, 12, and 14). Anthropic’s stance on model escape is basically like we’ll handle that in ASL-4 — this is reasonable and better than not noticing the threat but they’ve said ~nothing specific about escape/control yet.
I worry that quotes like
red-teaming should confirm that models can’t cause harm quickly enough to evade detection.
are misleading in the context of escape (from all labs, not just Anthropic). When defining ASL-4, Anthropic will need to clarify whether this includes scheming/escape and how safety could be “confirm[ed]” — I think safety properties will need high-effort control evals, not just red-teaming.
New: Anthropic lists the board members and LTBT members on a major page (rather than a blogpost with a specific date), which presumably means they’ll keep it up to date and thus we’ll quickly learn of changes, hooray:
Anthropic Board of Directors
Dario Amodei, Daniela Amodei, Yasmin Razavi, and Jay Kreps.LTBT Trustees
Neil Buddy Shah, Kanika Bahl, and Zach Robinson.
(We already knew these are the humans.)
In December 2023, Jason Matheny stepped down from the Trust to preempt any potential conflicts of interest that might arise with RAND Corporation’s policy-related initiatives. Paul Christiano stepped down in April 2024 to take a new role as the Head of AI Safety at the U.S. AI Safety Institute. Their replacements will be elected by the Trustees in due course.
Again, this is not news (although it hasn’t become well-known, I think) but I appreciate this public note.
Labs should give deeper model access to independent safety researchers (to boost their research)
Sharing deeper access helps safety researchers who work with frontier models, obviously.
Some kinds of deep model access:
Helpful-only version
Fine-tuning permission
Activations and logits access
[speculative] Interpretability researchers send code to the lab; the lab runs the code on the model; the lab sends back the results
See Shevlane 2022 and Bucknall and Trager 2023.
A lab is disincentivized from sharing deep model access because it doesn’t want headlines about how researchers got its model to do scary things.
It has been suggested that labs are also disincentivized from sharing because they want safety researchers to want to work at the labs and sharing model access with independent researchers make those researchers not need to work at the lab. I’m skeptical that this is real/nontrivial.
Labs should limit some kinds of access to avoid effectively leaking model weights. But sharing limited access with a moderate number of safety researchers seems very consistent with keeping control of the model.
This post is not about sharing with independent auditors to assess risks from a particular model.
@Buck suggested I write about this but I don’t have much to say about it. If you have takes—on the object level or on what to say in a blogpost on this topic—please let me know.
The LTBT, whose members have no equity in the company, currently elects one out of the board’s five members. But that number will rise to two out of five this July, and then to three out of five this November.
This is encouraging and makes me not care anymore about seeing the “milestones.” (And it explains why OP/investors/whoever didn’t bother to replace Luke.) My concerns about investors’ power over the Trust remain.
Also:
The LTBT’s first five members were picked by Anthropic’s executives for their expertise in three fields that the company’s co-founders felt were important to its mission: AI safety, national security, and social enterprise. Among those selected were Jason Matheny, CEO of the RAND corporation, Kanika Bahl, CEO of development nonprofit Evidence Action, and AI safety researcher Paul Christiano. [The other two were Neil Buddy Shah of the Clinton Health Access Initiative and formerly GiveWell and Zach Robinson of CEA and EV] (Christiano resigned from the LTBT prior to taking a new role in April leading the U.S. government’s new AI Safety Institute, he said in an email. His seat has yet to be filled.)
From this we can infer that the other four Trustees remain on the Trust, which is weak good news. [Edit: nope, Matheny left months ago due to potential (appearance of) conflict of interest. It’s odd that this article doesn’t mention that. As of May 31, Christiano and Matheny have not yet been replaced. It is maybe quite concerning if they—the two AI safety experts—are gone and not replaced by AI safety experts, and the Trust is putting non-AI-safety people on the board. Also I’m disappointed that Anthropic didn’t cause me to know this before.]
Also:
Amazon and Google, he says, do not own voting shares in Anthropic, meaning they cannot elect board members and their votes would not be counted in any supermajority required to rewrite the rules governing the LTBT. (Holders of Anthropic’s Series B stock, much of which was initially bought by the defunct cryptocurrency exchange FTX, also do not have voting rights, Israel says.)
Google and Amazon each own less than 15% of Anthropic, according to a person familiar with the matter.
So then the question is: who does own voting shares; how are voting shares distributed? and how can this change in the future?
(Also we just got two more examples of Anthropic taking credit for the Trust, and one of the articles even incorrectly says “power ultimately lies with a small, unaccountable group.”)
What are the best things—or some good things—MIRI comms has done or published in 2024?
I bet the timing is a coincidence or due to internal questions/pressure, not PR concerns. Regardless I should ask someone at Anthropic how this post was received within Anthropic.
The plan was for the Trust to elect a fifth board member and also eventually replace Luke and Daniela. I totally believe Anthropic that Luke’s departure was unrelated to Jay’s arrival and generally non-suspicious. [Edit: but I do wish he’d been replaced with a safety-focused board member. My weak impression is that OP has the right to fill that seat until the Trust does; probably OP wants to distance itself from Anthropic but just giving up a board seat seems like a bad call.]
Possibly he didn’t just mean technically difficult. And possibly Politico took this out of context. But I agree this quote is bad and clarification would be nice.
Jay Kreps, co-founder and CEO of Confluent, has joined Anthropic’s Board of Directors. . . . Jay was appointed to the board by Anthropic’s Long-Term Benefit Trust. . . . Separately, Luke Muehlhauser has decided to step down from his Board role to focus on his work at Open Philanthropy.
I’m glad that the Trust elected a board member.
I still really want to know whether this happened on schedule.
I’m interested in what will happen to Luke’s seat — my guess is that the Trust’s next appointment will fill it.
This is not relevant to my thesis: maybe the Trust can be overruled or abrogated by stockholders.
I agree that the Trust has some oversight over the RSP:
[Anthropic commitments:]
Share results of ASL evaluations promptly with Anthropic’s governing bodies, including the board of directors and LTBT, in order to sufficiently inform them of changes to our risk profile.
Responsible Scaling Officer. There is a designated member of staff responsible for ensuring that our Responsible Scaling Commitments are executed properly. Each quarter, they will share a report on implementation status to our board and LTBT, explicitly noting any deficiencies in implementation. They will also be responsible for sharing ad hoc updates sooner if there are any substantial implementation failures.
(This is nice but much less important than power over board seats.)
Minor remark, inspired (but not necessarily endorsed) by Buck:
The whole point of the Trust is to be able to act contrary to the interests of massively incentivized stakeholders. This is fundamentally a hard task, and it would be easy for the Trust Agreement to leave the Trust disempowered for practical purposes even if the people who wrote it weren’t trying to sabotage it. And as we saw with OpenAI, it’s dangerous to assume that the de facto power structures in an AI company match what’s on paper.
(This post is about a sharper, narrower concern — that if you read the relevant document you’d immediately conclude that the Trust has no real power.)
I also wonder about other aspects of Anthropic’s relationship with its stockholders:
Anthropic’s obligations to its stockholders & stockholders’ power over Anthropic
What its PBC status entails
Whether its investors have any special arrangements distinct from normal investment in a PBC
How stockholders can replace board members
How shares (or voting power) are currently distributed between Amazon, Google, OP, and others.
Thanks.
They were changing around their deployment system, infra, etc. We wanted uptime and throughput. Big pain in the ass to keep the model up (with proper access control) while they were overhauling stuff. Furthermore, anthropic and METR kept changing points of contact (rapidly growing teams).
This sounds like an unusual source of difficulty. Some Anthropic statements have suggested that sharing is hard in general. I hope it is just stuff like this. [Edit: possibly those statements were especially referring to deep model access or high-touch support. But then they don’t explain the labs’ lack of more basic sharing.]
New page on AI companies’ policy advocacy: https://ailabwatch.org/resources/company-advocacy/.
This page is the best collection on the topic (I’m not really aware of others), but I decided it’s low-priority and so it’s unpolished. If a better version would be helpful for you, let me know to prioritize it more.