My current model is that Anthropic has illegitimately taken up the flag of “big ML place that the people who want to avoid race dynamics go to” while basically doing most of the relevant work to be competitive, continuing to do tons of things that enhance race dynamics (e.g. being for-profit, aggressive fundraising rounds, massive staff size of just good engineers i.e. not the sort of x-risk-conscious people who would gladly stop their work if the leadership thought it was getting too close to AGI, developing cutting edge language models, etc), and the leadership basically having pretty poor models of alignment (e.g. I don’t know that any Anthropic staff are doing any work to implement experiments on even Paul’s ideas; I have also heard people who know them saying that they expect engineering leadership would not feel very concerned if their systems showed signs of deception [EDIT: see bottom of comment for update]), and broadly having little of the reasons to trust their judgment on these questions in the way I trust a few other people like Eliezer or Paul or John or Gwern — given that Dario/Daniela have written exceedingly little of their models and plans publicly, and from my occasional conversations with insiders literally don’t have plans other that don’t sound to me like “get as close to AGI as possible and then pause and do research with those AIs, probably this will work out fine”.
Might be wrong of course, but no way really to know, given the aforementioned lack of detailed explanations (or any really) of their views on the alignment problem and how to solve it.
That said, cybersecurity does sound interesting and potentially something that will extend timelines if it prevents foreign actors from stealing data, so could still be worth it (though Anthropic just got funding from Google, and if that relationship sustains, I presume that they will have access to Google’s cyberecurity folks? I honestly don’t know how competent they will be — whether they’re nailing it or will have given up on state-actor-level security). Also attempting to influence the leadership positively sounds potentially worthwhile.
Added: I recently heard a third-hand story of someone working there on scaling-up who said “I just think Dario’s plan is good”. When asked what Dario’s plan was, they said “Well, I think what I mean is that I just trust him to do the right thing when the time comes.” If this is accurate, then from my perspective, this is worse than Dario having a bad plan, because you can at least give a counter-argument to a bad plan. Someone with no plan doesn’t even have something to discuss. If you don’t have a plan, you can’t notice when it doesn’t match up with reality.
Edit: I PM’d with Evan, after his comment below, and it now seems to me that in the last 1.5 years Anthropic leadership has probably updated toward being more concerned about deceptive models.
massive staff size of just good engineers i.e. not the sort of x-risk-conscious people who would gladly stop their work if the leadership thought it was getting too close to AGI
From my interactions with engineers at Anthropic so far, I think is a mischaracterization. I think the vast majority are in fact pretty x-risk conscious and my guess is that in fact if leadership said stop people would be happy to stop.
engineering leadership would not feel very concerned if their systems showed signs of deception
I’ve had personal conversations with Anthropic leadership about this and can confirm that it is definitely false; I think Jared and Dario would be quite concerned if they saw deceptive alignment in their models.
That’s good to hear you think that! I’d find it quite helpful to know the results of a survey to the former effect, of the (40? 80?) ML engineers and researchers there, anonymously answering a question like “Insofar as your job involves building large language models, if Dario asked you to stop your work for 2 years while still being paid your salary, how likely would you be to do so (assume the alternative is being fired)? (1-10, Extremely Unlikely, Extremely Likely)” and the same question but “Condition on it looking to you like Anthropic and OpenAI are both 1-3 years from building AGI”. I’d find that evidence quite informative. Hat tip to Habryka for suggesting roughly this question to me a year ago.
(I’m available and willing to iterate on a simple survey to that effect if you are too, and can do some iteration/user-testing with other people.)
(I’ll note that if the org doubles in size every year or two then… well, I don’t know how many x-risk conscious engineers you’ll get, or what sort of enculturation Anthropic will do in order to keep the answer to this up at 90%+.)
Regarding the latter, I’ve DM’d you about the specifics.
Consider writing up some of these impressions publicly. I would have talked to a couple people at the org before joining, but as someone who is almost completely disconnected from the “rationalist scene” physically, all I have to go on are what people say about the org on the internet. I don’t really have access to the second- or third-hand accounts that you probably have.
The signals I can remember updating against over the past were something like:
A few offhand tweets, before I stopped reading twitter, by EY, explaining that he was more-impressed-than-expected with their alignment research.
Some comments on LW I can’t remember alleging that Anthropic was started via an exodus of OpenAI’s most safety-conscious researchers.
Their website and general policy of not publishing capabilities research by default.
The identities and EA-affiliations of the funders. Jaan Tallinnn seems like a nice person. SBF was not a good person, but his involvement lets me infer certain things about their pitch/strategy that I can’t otherwise.
This post by evhub and just the fact that they have former MIRI researchers joining the team at all. Didn’t even remember this part until he commented.
In retrospect maybe these are some pretty silly things to base an opinion of an AGI organization on. But I guess you could say their marketing campaign was successful, and my cautious opinion was that they were pretty sincere and effective.
My current model is that Anthropic has illegitimately taken up the flag of “big ML place that the people who want to avoid race dynamics go to” while basically doing most of the relevant work to be competitive, continuing to do tons of things that enhance race dynamics (e.g. being for-profit, aggressive fundraising rounds, massive staff size of just good engineers i.e. not the sort of x-risk-conscious people who would gladly stop their work if the leadership thought it was getting too close to AGI, developing cutting edge language models, etc), and the leadership basically having pretty poor models of alignment (e.g. I don’t know that any Anthropic staff are doing any work to implement experiments on even Paul’s ideas; I have also heard people who know them saying that they expect engineering leadership would not feel very concerned if their systems showed signs of deception [EDIT: see bottom of comment for update]), and broadly having little of the reasons to trust their judgment on these questions in the way I trust a few other people like Eliezer or Paul or John or Gwern — given that Dario/Daniela have written exceedingly little of their models and plans publicly, and from my occasional conversations with insiders literally don’t have plans other that don’t sound to me like “get as close to AGI as possible and then pause and do research with those AIs, probably this will work out fine”.
Might be wrong of course, but no way really to know, given the aforementioned lack of detailed explanations (or any really) of their views on the alignment problem and how to solve it.
That said, cybersecurity does sound interesting and potentially something that will extend timelines if it prevents foreign actors from stealing data, so could still be worth it (though Anthropic just got funding from Google, and if that relationship sustains, I presume that they will have access to Google’s cyberecurity folks? I honestly don’t know how competent they will be — whether they’re nailing it or will have given up on state-actor-level security). Also attempting to influence the leadership positively sounds potentially worthwhile.
Added: I recently heard a third-hand story of someone working there on scaling-up who said “I just think Dario’s plan is good”. When asked what Dario’s plan was, they said “Well, I think what I mean is that I just trust him to do the right thing when the time comes.” If this is accurate, then from my perspective, this is worse than Dario having a bad plan, because you can at least give a counter-argument to a bad plan. Someone with no plan doesn’t even have something to discuss. If you don’t have a plan, you can’t notice when it doesn’t match up with reality.
Edit: I PM’d with Evan, after his comment below, and it now seems to me that in the last 1.5 years Anthropic leadership has probably updated toward being more concerned about deceptive models.
From my interactions with engineers at Anthropic so far, I think is a mischaracterization. I think the vast majority are in fact pretty x-risk conscious and my guess is that in fact if leadership said stop people would be happy to stop.
I’ve had personal conversations with Anthropic leadership about this and can confirm that it is definitely false; I think Jared and Dario would be quite concerned if they saw deceptive alignment in their models.
That’s good to hear you think that! I’d find it quite helpful to know the results of a survey to the former effect, of the (40? 80?) ML engineers and researchers there, anonymously answering a question like “Insofar as your job involves building large language models, if Dario asked you to stop your work for 2 years while still being paid your salary, how likely would you be to do so (assume the alternative is being fired)? (1-10, Extremely Unlikely, Extremely Likely)” and the same question but “Condition on it looking to you like Anthropic and OpenAI are both 1-3 years from building AGI”. I’d find that evidence quite informative. Hat tip to Habryka for suggesting roughly this question to me a year ago.
(I’m available and willing to iterate on a simple survey to that effect if you are too, and can do some iteration/user-testing with other people.)
(I’ll note that if the org doubles in size every year or two then… well, I don’t know how many x-risk conscious engineers you’ll get, or what sort of enculturation Anthropic will do in order to keep the answer to this up at 90%+.)
Regarding the latter, I’ve DM’d you about the specifics.
Consider writing up some of these impressions publicly. I would have talked to a couple people at the org before joining, but as someone who is almost completely disconnected from the “rationalist scene” physically, all I have to go on are what people say about the org on the internet. I don’t really have access to the second- or third-hand accounts that you probably have.
The signals I can remember updating against over the past were something like:
A few offhand tweets, before I stopped reading twitter, by EY, explaining that he was more-impressed-than-expected with their alignment research.
Some comments on LW I can’t remember alleging that Anthropic was started via an exodus of OpenAI’s most safety-conscious researchers.
Their website and general policy of not publishing capabilities research by default.
The identities and EA-affiliations of the funders. Jaan Tallinnn seems like a nice person. SBF was not a good person, but his involvement lets me infer certain things about their pitch/strategy that I can’t otherwise.
This post by evhub and just the fact that they have former MIRI researchers joining the team at all. Didn’t even remember this part until he commented.
In retrospect maybe these are some pretty silly things to base an opinion of an AGI organization on. But I guess you could say their marketing campaign was successful, and my cautious opinion was that they were pretty sincere and effective.