A drastic table-flip-like action a safety-conscious frontier lab could take: Burn their company to the ground
At a specific chosen moment, all employees are fired, the leadership steps down, all intellectual property+model weights are deleted and infrastructure is de-deployed/”demolished”, and this is publicly announced, together with the public message that it was too dangerous to continue, and an urging of other company leaders to do the same.
(I don’t think any lab is planning on doing this, but someone could.)
IANAL but I believe it would be legal for OpenAI (which is a nonprofit) and Anthropic (which is a public benefit corporation).
I don’t think it is known whether it would be legal for a for-profit because there’s no precedent.
“Sorry, we wish we could’ve not killed everyone, but we had to uphold our fiduciary duty” is a really weak defense.
There are some kinds of lawbreaking that I would not endorse (e.g. violence) but I have no hard line against violating fiduciary duty.
(In general, you should abide by your fiduciary duty, but there are some ethical prescriptions that weigh more heavily, and I think this is widely recognized in many contexts. For example whistleblowers who expose companies’ unethical behavior are violating their fiduciary duty because exposing bad behavior reduces profit.)
It’s not obvious to me that an illegal act of this nature would complete successfully. Why wouldn’t a company that did such a thing not be court-ordered to reassemble? How much time do you think humanity would gain from this?
I think this would be the correct thing to do for an appropriately safety-minded AI company. This is a strong signal to governments to wake up to AI x-risk, and it puts pressure on other companies to take safety more seriously.
Given (1) uncertainty about timelines/takeoff speeds and (2) uncertainty about how long it would take for the good effects to pay off, I think a safety-minded AI company should shut down immediately. (I think a year ago would have also been a good time.)
Another thing a company could do is completely stop working on capabilities, and dedicate 100% of resources to alignment research / other AI safety activities until they run out of money. For a for-profit this would violate its fiduciary duty but I think (IANAL) OpenAI and Anthropic would be on solid legal ground because they’re not pure for-profits. I also think there’s a (correct) argument to be made that decreasing your shareholders’ risk of dying is consistent with fiduciary duty (although I kind of doubt courts would go for that).
A drastic table-flip-like action a safety-conscious frontier lab could take: Burn their company to the ground
At a specific chosen moment, all employees are fired, the leadership steps down, all intellectual property+model weights are deleted and infrastructure is de-deployed/”demolished”, and this is publicly announced, together with the public message that it was too dangerous to continue, and an urging of other company leaders to do the same.
(I don’t think any lab is planning on doing this, but someone could.)
Would this be legal for any currently-existing lab to do? I doubt it, but I am not a lawyer.
IANAL but I believe it would be legal for OpenAI (which is a nonprofit) and Anthropic (which is a public benefit corporation).
I don’t think it is known whether it would be legal for a for-profit because there’s no precedent.
“Sorry, we wish we could’ve not killed everyone, but we had to uphold our fiduciary duty” is a really weak defense.
There are some kinds of lawbreaking that I would not endorse (e.g. violence) but I have no hard line against violating fiduciary duty.
(In general, you should abide by your fiduciary duty, but there are some ethical prescriptions that weigh more heavily, and I think this is widely recognized in many contexts. For example whistleblowers who expose companies’ unethical behavior are violating their fiduciary duty because exposing bad behavior reduces profit.)
xAI is apparently also a PBC, which I didn’t know beforehand.
That would make it more effective — very expensive signaling is the very best kind…
It’s not obvious to me that an illegal act of this nature would complete successfully. Why wouldn’t a company that did such a thing not be court-ordered to reassemble? How much time do you think humanity would gain from this?
Prior related discussion, for completeness:
https://www.lesswrong.com/posts/8vgi3fBWPFDLBBcAx/planning-for-extreme-ai-risks#2_3__Outcome__3__Self_destruction
I think this would be the correct thing to do for an appropriately safety-minded AI company. This is a strong signal to governments to wake up to AI x-risk, and it puts pressure on other companies to take safety more seriously.
Given (1) uncertainty about timelines/takeoff speeds and (2) uncertainty about how long it would take for the good effects to pay off, I think a safety-minded AI company should shut down immediately. (I think a year ago would have also been a good time.)
Another thing a company could do is completely stop working on capabilities, and dedicate 100% of resources to alignment research / other AI safety activities until they run out of money. For a for-profit this would violate its fiduciary duty but I think (IANAL) OpenAI and Anthropic would be on solid legal ground because they’re not pure for-profits. I also think there’s a (correct) argument to be made that decreasing your shareholders’ risk of dying is consistent with fiduciary duty (although I kind of doubt courts would go for that).
Relevant Manifold market.