Anthropic’s ownership is like 10% founder (of which 17% will be spent on good AI safety stuff)
On a dollar-weighted basis, Anthropic shareholders overwhelmingly have the sorts of views about superintelligence that lead them to believe that accelerating AI capabilities is a good idea, which makes me think their donations will mostly not be very good. My guess is the vast majority of Anthropic-sourced donations will go to the sort of work that AI companies were already doing internally anyway—basically, inoffensive safety research that’s unlikely to actually prevent extinction.
EDIT: This comment was supposed to make a narrow point but I didn’t make that sufficiently clear. Michael doesn’t see this as very load-bearing to his overall argument in his third comment in the thread.
Anthropic shareholders overwhelmingly have the sorts of views about superintelligence that lead them to believe that accelerating AI capabilities is a good idea
I think Anthropic leadership believe that slightly slowing down AI capabilities progress is net-good for society. I’m focused on Anthropic leadership here because they’ve pledged much of their money for donations[1] and Claude says that Anthropic leadership owns ~14% of Anthropic (over $100B).
We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology. The Anthropic Institute will conduct research—in collaboration with many others—and take actions to help build the systems that a credible slowdown or pause would require. These systems would enable frontier AI developers to verify that others globally have actually stopped or slowed, and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret.If such systems existed, we expect that we would slow down or temporarily pause, if other developers at or near the frontier also did so in a verifiable manner.
There’s admittedly a bit of ambiguity in their positions of whether slowdowns are good in both the first and last sentences here.[2] Still, unless this ambiguity is deliberate, I read this as them thinking that a slight slowdown is good.
My sense from Dario based on some interviews and writings is that he expects AGI soon and he expects that leadership are not ready for the exponential.
It’s also possible that these statements are intentional lies / deceptions, but I don’t find that likely. I find Anthropic leadership’s overall story of “we think other companies are being irresponsible but we think p(x-risk) is low relative to the average LWer so we’ll try to build AGI” to make sense in combination with believing “TAI coming somewhat more slowly is better for society as a whole” especially given how fast Anthropic timelines are. Even if they’re optimistic as a whole about the benefits of TAI, it still seems right to prefer it to come out more slowly. Also, they’d prefer for TAI to be developed after Trump’s term ends if they could pick.
Is the first sentence just in favor of option value or a general slowdown? Is the last sentence just conditional: other labs are only likely to slowdown/pause if they realize some unforeseen risks or if a global treaty is enforced.
I don’t know what true feelings live in their hearts, but behaviorally speaking, Anthropic leadership’s behavior is not consistent with what I’d predict from someone who wants to slow down AI. It is consistent with what I’d predict from someone who wants to placate AI safety people while continuing to race ahead.
I liked Nate Soares’ recent tweets about this (1, 2, 3). Excerpt:
Wikipedia spends more effort convincing you to donate $2 than AI companies spend galvanizing the world to stop the race. There’s a real inconsistency between what they say and how they behave. This mealy-mouthed behavior incurs real costs.
Fair. I think I could have phrased it better, but my point is more like Anthropic doesn’t seem to care too much about AI development speeds (or their impact on accelerating it) but insofar as they care, they’d think its better for society to have slightly slower societal AI progress. I think this is also consistent with their views.
I do agree that Anthropic’s accelerating timelines so much (along with some dishonest behavior) was bad and Dario is very overoptimistic about the benefits of AI.
Ok, my last comment was not that relevant then. What I would say instead is: I believe the most important thing to be doing at current margins is trying to pause AI, and I do not expect the biggest Anthropic donors to spend their money on trying to pause AI. If “Anthropic doesn’t seem to care too much about AI development speeds”, then they’ll probably spend their donation money on other things like technical alignment research, right?
Maybe in my original comment I shouldn’t have said they “believe that accelerating AI capabilities is a good idea” because that’s not quite right (it’s more like “believe that working on AI capabilities is okay”) and, more importantly, it’s not the relevant bit of info—the relevant thing is that they’re much more optimistic than I am about their chances of solving AI alignment or otherwise successfully navigating transformative AI without pausing, which I think will cause them to spend money in ways that aren’t very useful according to my beliefs. And not just that we disagree on that particular issue, but that they are not thinking clearly about AI risk, and I expect this will cause them to spend money in highly suboptimal ways even if I can’t predict exactly what they will want to spend money on.
On a dollar-weighted basis, Anthropic shareholders overwhelmingly have the sorts of views about superintelligence that lead them to believe that accelerating AI capabilities is a good idea, which makes me think their donations will mostly not be very good. My guess is the vast majority of Anthropic-sourced donations will go to the sort of work that AI companies were already doing internally anyway—basically, inoffensive safety research that’s unlikely to actually prevent extinction.
EDIT: This comment was supposed to make a narrow point but I didn’t make that sufficiently clear. Michael doesn’t see this as very load-bearing to his overall argument in his third comment in the thread.
I think Anthropic leadership believe that slightly slowing down AI capabilities progress is net-good for society. I’m focused on Anthropic leadership here because they’ve pledged much of their money for donations[1] and Claude says that Anthropic leadership owns ~14% of Anthropic (over $100B).
From Anthropic’s When AI builds itself
There’s admittedly a bit of ambiguity in their positions of whether slowdowns are good in both the first and last sentences here.[2] Still, unless this ambiguity is deliberate, I read this as them thinking that a slight slowdown is good.
My sense from Dario based on some interviews and writings is that he expects AGI soon and he expects that leadership are not ready for the exponential.
It’s also possible that these statements are intentional lies / deceptions, but I don’t find that likely. I find Anthropic leadership’s overall story of “we think other companies are being irresponsible but we think p(x-risk) is low relative to the average LWer so we’ll try to build AGI” to make sense in combination with believing “TAI coming somewhat more slowly is better for society as a whole” especially given how fast Anthropic timelines are. Even if they’re optimistic as a whole about the benefits of TAI, it still seems right to prefer it to come out more slowly. Also, they’d prefer for TAI to be developed after Trump’s term ends if they could pick.
Although it remains to be seen how much actually is donated
Is the first sentence just in favor of option value or a general slowdown? Is the last sentence just conditional: other labs are only likely to slowdown/pause if they realize some unforeseen risks or if a global treaty is enforced.
I don’t know what true feelings live in their hearts, but behaviorally speaking, Anthropic leadership’s behavior is not consistent with what I’d predict from someone who wants to slow down AI. It is consistent with what I’d predict from someone who wants to placate AI safety people while continuing to race ahead.
I liked Nate Soares’ recent tweets about this (1, 2, 3). Excerpt:
Fair. I think I could have phrased it better, but my point is more like Anthropic doesn’t seem to care too much about AI development speeds (or their impact on accelerating it) but insofar as they care, they’d think its better for society to have slightly slower societal AI progress. I think this is also consistent with their views.
I do agree that Anthropic’s accelerating timelines so much (along with some dishonest behavior) was bad and Dario is very overoptimistic about the benefits of AI.
Ok, my last comment was not that relevant then. What I would say instead is: I believe the most important thing to be doing at current margins is trying to pause AI, and I do not expect the biggest Anthropic donors to spend their money on trying to pause AI. If “Anthropic doesn’t seem to care too much about AI development speeds”, then they’ll probably spend their donation money on other things like technical alignment research, right?
Maybe in my original comment I shouldn’t have said they “believe that accelerating AI capabilities is a good idea” because that’s not quite right (it’s more like “believe that working on AI capabilities is okay”) and, more importantly, it’s not the relevant bit of info—the relevant thing is that they’re much more optimistic than I am about their chances of solving AI alignment or otherwise successfully navigating transformative AI without pausing, which I think will cause them to spend money in ways that aren’t very useful according to my beliefs. And not just that we disagree on that particular issue, but that they are not thinking clearly about AI risk, and I expect this will cause them to spend money in highly suboptimal ways even if I can’t predict exactly what they will want to spend money on.