Something that I’ve continually found surprising about AI Safety as opposed to working at Nvidia / Apple is that frontier labs (now potentially multi-trillion dollar valuation for profit tech companies) are not treated like you’d treat any other multi-trillion dollar valuation for profit tech companies by default.
I’m confused about much of the discussion on this post being about whether Anthropic has done “net good”.
The post is very specifically a deep dive into the fact that Anthropic, like any other company, should not have their leadership’s statements taken at face value. IMO this is a completely unrealistic way to treat companies in any field, and it’s a bit frustrating to see the rationalist presumption of good faith extended over and over by default in contexts where it’s so incredibly exploitable.
Again this is not a specific criticism of Anthropic, if a new lab starts tomorrow promising to build Safe SuperIntelligence for example, we should not assume that we can trust all their leadership’s statements until they’ve mislead people publicly a few times and someone has a deep dive comprehensively documenting it.
but wanted to note that I think it applies across labs, for example, OpenAI (and Anthropic) both released a batch of misalignment incidents today. Sydney correctly pointed out that this is the third time they’ve dumped these incidents on a Friday:
Kai (OpenAI) then responded:
This is obviously intentional, and is standard practice for large companies with comms teams who want to bury bad news:
While it’s technically true that “there could be innocent good faith explanations” and “corporations aren’t a monolith and have diverse views internally”, these are completely unreasonable defaults to have about large tech companies.
It often seems extremely taboo to assume that anyone is operating out of anything other than good faith or some other variant of mistake theory. This does not feel like a serious way to approach dealing with large tech companies and IMO it is constantly leveraged against people in AI Safety when they interact with frontier labs.
It often seems extremely taboo to assume that anyone is operating out of anything other than good faith or some other variant of mistake theory. This does not feel like a serious way to approach dealing with large tech companies and IMO it is constantly leveraged against people in AI Safety when they interact with frontier labs.
I agree and try to call this out on Twitter when I can. It’s incredibly frustrating to see the obvious bad faith from different actors in the field, yet the virtue signalling for the principle of charity within the community makes them highly-susceptible to be continually duped and taken advantage of. It’s seen as bad to call people out without hard evidence and loonie to include larger context and patterns of behaviour as highly relevant in interpreting a situation.
Unfortunately, this can be a hard thing to navigate because unless you have hard evidence for what a lab is doing is straightforwardly malicious, it becomes too easy for lab employees to start developing a tribal mentality where increasingly all critics have “[AI company] derangement syndrome”.
Worth remembering that OpenAI started as a nonprofit, and Anthropic branded itself as basically a nonprofit that was structured as a PBC for convenience. But yes I agree with your take.
I do think that lab researchers currently have influence over company policy and can steer it in positive directions. But if Anthropic and OAI IPO and generally become more like for-profit tech companies, we should expect this influence to wane, even before they are automated away.
If I was operating in complete good faith for the betterment of humanity, would I still not want to cynically manipulate public opinion by releasing bad news on Friday afternoons, to thereby gain power I can use for good?
Given how similarly good and evil look from the outside, maybe we should always just assume the worst.
My impression is that public comms from third party orgs are forced to be very friendly-sounding like this because third party orgs depend on labs for privileged access. I’d be surprised if the actual thinking is so uncritical
Lots of actions that these labs are taking show that it’s at least an oversimplification to treat them like any other trillion dollar valuation company.
Maybe most importantly, one would make the mistake of treating CEO’s and employee’s warnings of existential risks as some kind of marketing ploy.
It’s also pretty clear that OpenAI grants their employees pretty unusual freedoms to criticize the company publicly (just compare to Google). In this context, I’d notice that an OpenAI employee, especially if they are on the safety team, imply that they are not trying to bury it, and update towards it really not being “obviously intentional” that this is an attempt to bury it.
It’s also pretty clear that OpenAI grants their employees pretty unusual freedoms to criticize the company publicly (just compare to Google)
Examples? People keep saying this, but I don’t think I’ve ever seen any actually meaningful critique of OpenAI by standing OpenAI employees. Dan Selsam’s statement and some of the things @leogao said are the closest, but even there, as far as I remember there are no concrete specific criticisms of what OpenAI does. At most there are some insinuations that OpenAI should do better in the future, and passive-voice discussions of bad AI-race dynamics that don’t attribute any responsibility to any specific entity. It all seems very much within the boundaries of “permitted discontent”; the kind of culture you’d cultivate if you wanted to create an impression of unusually free thought, without actually risking people saying something inconvenient.
The real critiques only come out after the employee quits or is fired, as usual.
i’m surprised that you don’t think i’ve ever said any concrete specific criticisms. i feel like i’ve made many such criticisms; i asked chatgpt for examples and it pulled up posts i made about the dow contract, bad safety incentives, and net harmfulness of much of the prosaic alignment research.
I’m under the impression that the companies have no legal obligation to release public writeups of incidents and are choosing to do so voluntarily? In that case it would seem odd if they were also trying to bury those releases?
It’s a bit annoying because actions of hypothetical good faith and bad faith versions of the frontier companies might look really similar to each other. I overall feel confused how to think about this
The Ten People On The Inside likely should prioritize things that capabilities teams aren’t going to do anyway and people considering joining labs with similar motivations should likely think about whether they’re sure they’re going to be able to do work that wouldn’t otherwise be done for capabilities reasons. The thing that’s top of mind here is reward hacking, which is increasingly becoming a capabilities problem.[1] To the extent research is done here, it seems pretty important to avoid the failure mode of “repeatedly iterate on environments and observable behavior”[2] which is the kind of thing I would expect by default for capabilities reasons.
A good example of research on reward hacking that I don’t think would’ve necessarily gotten done “by default for capabilities reasons” is inoculation prompting and similar investigations
This can take the form of specific safety training targeting the observable behavior, post-hoc environment fixes, or having an iteration loop between your pre-deployment automated alignment evals and your alignment training
A good example of research on reward hacking that I don’t think would’ve necessarily gotten done “by default for capabilities reasons” is inoculation prompting and similar investigations
I’ve seen the sentiment a few times that we don’t need solutions to the issues raised in Automating Alignment Is Harder Than You Think[1] , i.e. something like scalable oversight / weak-to-strong generalization. Has anyone made this case before? I’ve heard it often from people inside (and even outside) of labs, and my impression is that it’s load bearing for research prioritization with respect to which research directions to pursue and how to allocate researcher time.
Something that I’ve continually found surprising about AI Safety as opposed to working at Nvidia / Apple is that frontier labs (now potentially multi-trillion dollar valuation for profit tech companies) are not treated like you’d treat any other multi-trillion dollar valuation for profit tech companies by default.
I’ve made this point before as it comes up most often with Anthropic:
but wanted to note that I think it applies across labs, for example, OpenAI (and Anthropic) both released a batch of misalignment incidents today. Sydney correctly pointed out that this is the third time they’ve dumped these incidents on a Friday:
Kai (OpenAI) then responded:
This is obviously intentional, and is standard practice for large companies with comms teams who want to bury bad news:
While it’s technically true that “there could be innocent good faith explanations” and “corporations aren’t a monolith and have diverse views internally”, these are completely unreasonable defaults to have about large tech companies.
It often seems extremely taboo to assume that anyone is operating out of anything other than good faith or some other variant of mistake theory. This does not feel like a serious way to approach dealing with large tech companies and IMO it is constantly leveraged against people in AI Safety when they interact with frontier labs.
I agree and try to call this out on Twitter when I can. It’s incredibly frustrating to see the obvious bad faith from different actors in the field, yet the virtue signalling for the principle of charity within the community makes them highly-susceptible to be continually duped and taken advantage of. It’s seen as bad to call people out without hard evidence and loonie to include larger context and patterns of behaviour as highly relevant in interpreting a situation.
Unfortunately, this can be a hard thing to navigate because unless you have hard evidence for what a lab is doing is straightforwardly malicious, it becomes too easy for lab employees to start developing a tribal mentality where increasingly all critics have “[AI company] derangement syndrome”.
Worth remembering that OpenAI started as a nonprofit, and Anthropic branded itself as basically a nonprofit that was structured as a PBC for convenience. But yes I agree with your take.
I do think that lab researchers currently have influence over company policy and can steer it in positive directions. But if Anthropic and OAI IPO and generally become more like for-profit tech companies, we should expect this influence to wane, even before they are automated away.
If I was operating in complete good faith for the betterment of humanity, would I still not want to cynically manipulate public opinion by releasing bad news on Friday afternoons, to thereby gain power I can use for good?
Given how similarly good and evil look from the outside, maybe we should always just assume the worst.
My impression is that public comms from third party orgs are forced to be very friendly-sounding like this because third party orgs depend on labs for privileged access. I’d be surprised if the actual thinking is so uncritical
Lots of actions that these labs are taking show that it’s at least an oversimplification to treat them like any other trillion dollar valuation company.
Maybe most importantly, one would make the mistake of treating CEO’s and employee’s warnings of existential risks as some kind of marketing ploy.
It’s also pretty clear that OpenAI grants their employees pretty unusual freedoms to criticize the company publicly (just compare to Google). In this context, I’d notice that an OpenAI employee, especially if they are on the safety team, imply that they are not trying to bury it, and update towards it really not being “obviously intentional” that this is an attempt to bury it.
Examples? People keep saying this, but I don’t think I’ve ever seen any actually meaningful critique of OpenAI by standing OpenAI employees. Dan Selsam’s statement and some of the things @leogao said are the closest, but even there, as far as I remember there are no concrete specific criticisms of what OpenAI does. At most there are some insinuations that OpenAI should do better in the future, and passive-voice discussions of bad AI-race dynamics that don’t attribute any responsibility to any specific entity. It all seems very much within the boundaries of “permitted discontent”; the kind of culture you’d cultivate if you wanted to create an impression of unusually free thought, without actually risking people saying something inconvenient.
The real critiques only come out after the employee quits or is fired, as usual.
i’m surprised that you don’t think i’ve ever said any concrete specific criticisms. i feel like i’ve made many such criticisms; i asked chatgpt for examples and it pulled up posts i made about the dow contract, bad safety incentives, and net harmfulness of much of the prosaic alignment research.
That’s fair; the DoW thing specifically is indeed very clear and concrete.
I’m under the impression that the companies have no legal obligation to release public writeups of incidents and are choosing to do so voluntarily? In that case it would seem odd if they were also trying to bury those releases?
One lens from which it’s not odd:
releasing at all helps X seem more responsible to safety conscious people so that Y
burying release helps X avoid consequences Y of admitting mistakes
X might be a frontier company or some faction within it, Y might be avoiding public or company protests, resignations, regulations, etc
It’s a bit annoying because actions of hypothetical good faith and bad faith versions of the frontier companies might look really similar to each other. I overall feel confused how to think about this
The Ten People On The Inside likely should prioritize things that capabilities teams aren’t going to do anyway and people considering joining labs with similar motivations should likely think about whether they’re sure they’re going to be able to do work that wouldn’t otherwise be done for capabilities reasons. The thing that’s top of mind here is reward hacking, which is increasingly becoming a capabilities problem.[1] To the extent research is done here, it seems pretty important to avoid the failure mode of “repeatedly iterate on environments and observable behavior”[2] which is the kind of thing I would expect by default for capabilities reasons.
A good example of research on reward hacking that I don’t think would’ve necessarily gotten done “by default for capabilities reasons” is inoculation prompting and similar investigations
This can take the form of specific safety training targeting the observable behavior, post-hoc environment fixes, or having an iteration loop between your pre-deployment automated alignment evals and your alignment training
Legible vs. Illegible AI Safety Problems somewhat resonates with this.
Why?
I’ve seen the sentiment a few times that we don’t need solutions to the issues raised in Automating Alignment Is Harder Than You Think[1] , i.e. something like scalable oversight / weak-to-strong generalization. Has anyone made this case before? I’ve heard it often from people inside (and even outside) of labs, and my impression is that it’s load bearing for research prioritization with respect to which research directions to pursue and how to allocate researcher time.
Or similar issues raised in https://joecarlsmith.com/2025/04/30/can-we-safely-automate-alignment-research/#5-evaluation-failures or https://openai.com/index/introducing-superalignment/
I was a bit surprised in Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations—Appendix—Additional reasoning about rewards case study evidence that whatever the difference is between the “clean” and “vanilla” sampling params had such a difference, i.e. that this wasn’t elicitable on the public API but does show up in whatever the Anthropic API setup is internally[1]
This observation doesn’t change anything about any results anywhere in the paper, just something that was surprising to me