Fair enough; we certainly paid much less than people could make in the private sector at CAIP, for essentially that reason. It’s good for nonprofit staff to have some skin in the game.
My suggestion to consider more competitive wages is mostly a response to Oliver suggesting that LTFF has had a serious and long-term challenge in hiring as many people as they would need to fully accomplish their mission.
Part of the distinction I try to draw in my sequence is that the median person at CSET or RAND is not “in politics” at all. They’re mostly researchers at think tanks, writing academic-style papers about what kinds of policies would be theoretically good for someone to adopt. Their work is somewhat more applied/concrete than the work of, e.g., a median political science professor at a state university, but not by a wide margin.
If you want political experts—and you should—you have to go talk to people who have worked on political campaigns, served in the government, or led advocacy organizations whose mission is to convince specific politicians to do specific things. This is not the same thing as a policy expert.
For what it’s worth, I do think OpenPhil and other large EA grantmakers should be hiring many more people. Hiring any one person too quickly is usually a mistake, but making sure that you have several job openings posted at any given time (each of which you vet carefully) is not.
I’m the author of the LW post being signal-boosted. I sincerely appreciate Oliver’s engagement with these critiques, and I also firmly disagree with his blanket dismissal of the value of “standard practices.”
As I argue in the 7th post in the linked sequence, I think OpenPhil and others are leaving serious value on the table by not adopting some of the standard grant evaluation practices used at other philanthropies, and I don’t think they can reasonably claim to have considered and rejected them—instead the evidence strongly suggests that they’re (a) mostly unaware of these practices due to not having brought in enough people with mainstream expertise, and (b) quickly deciding that anything that seems unfamiliar or uncomfortable “doesn’t make sense” and can therefore be safely ignored.
We have a lot of very smart people in the movement, as Oliver correctly points out, and general intelligence can get you pretty far in life, but Washington, DC is an intensely competitive environment that’s full of other very smart people. If you try to compete here with your wits alone while not understanding how politics works, you’re almost certainly going to lose.
These are important points, and I’m glad you’re bringing them up!
Is spending a lot of time to assess new grantmakers merely distressing (but still net positive in terms of extending your total grantmaking ability), or is it actually causing you to lose time in expectation? In other words, if you spend 40 hours recruiting and assessing candidates, does one of those candidates then go on to do 100+ hours of useful grantmaking work? Or is it more like 20 hours of useful grantmaking work?
How closely connected is the shortage of people willing to be full-time grantmakers with an expectation that grantmakers will already be fluent in technical AI safety when they start work? I could imagine that people who could otherwise be working for ARC or Anthropic would be very difficult to lure away full-time, but there’s an entire field of mainstream philanthropic foundations that mostly have full-time staff working on their grants. Could we hire some of those grantmakers full time to lend their general grantmaking expertise, evaluating things like budgets and org charts and performance targets, while relying on part-time advisors to provide technical expertise about the details of AI safety research? If not, why not?
What do you see as the most likely or most important negative consequence if grantmakers try to offer highly competitive salaries? Is this something that your funders have literally refused to pay for, or are you worried about being criticized for it (by whom? what consequences would follow from that criticism?), or does it just generally increase team members’ anxiety levels, or what exactly is the downside? I have definitely seen some of this paranoia you’re talking about, so it’s a real problem, but I wonder if it’s worth accepting the costs associated with paying highly competitive salaries in order to attract more and better people. It’s also worth noting that ‘highly competitive nonprofit salaries’ are still lower than ‘highly competitive tech salaries,’ probably by a factor of about 3. You can get top-notch grantmaking talent for much less than the price of top-notch computer engineering talent.
I mostly just got older and therefore calmer. I’ve crossed off most of the highest-priority items from my bucket list, so while I would prefer to continue living for a good long while, my personal death and/or defeat doesn’t seem so catastrophically bad anymore, and to cope with the loss of civilization/humanity I read a lot of history and sci-fi and anthropology and other works that help me zoom out and see that there has already been great loss and that while I do want to spend my resources fighting to reduce the risk of that loss, it’s not something I need to spend a lot of time or energy personally suffering over, especially not in advance. Worry is interest paid on trouble before it’s due.
Interesting thoughts; thanks for sharing, and for your work at CeSIA.
I’ve put some work into building coordination among US AI safety advocates, and it’s been somewhat helpful, but there are limits to how much we can expect discussions about coordination to lead to unified action because different organizations have different funders, different principles, and different interests. Merely sharing information about what different groups are working on will not spontaneously cause those groups to pick a single task and pivot to supporting it.
I suppose I was speaking too loosely—thank you for flagging that!
I don’t mean that it’s literally impossible to assess whether AI governance grants have been successful—only that doing so requires somewhat more deliberate effort than it does for most other types of grants, and that there is relatively less in the way of established infrastructure to support such measurements in the field of AI governance.
If you run an anti-malaria program, there’s a consensus about at least the broad strokes of what you’re supposed to measure (i.e., malaria cases), and you’ll get at least some useful information about that metric just from running your program and honestly recording what your program officers observe as they deliver medication. If your bed nets are radically reducing the incidence of malaria in your target population, then the people distributing those bed nets will probably notice. There is also an established literature on “experimental methods” for these kinds of interventions that tells us that we need to be taking measurements and how to do so and how to interpret them.
By contrast, if you’re slightly reducing the odds of an AI catastrophe, it’s not immediately obvious or agreed-upon what observable changes this ought to produce in the real world, and a grant funder isn’t very likely to notice those changes unless they specifically go and look for them. They’re also less likely to specifically go and look for them in an effective way, because the literature on experimental methods for politics is much less well-developed than the literature on experimental methods for public health.
My work so far has mostly been about doing the advocacy, rather than establishing better metrics to evaluate the impact of that advocacy. That said, in posts 1 and 7 of this sequence, I do suggest some starting points. I encourage funders to look at figures like the number of meetings had with politicians, the number of events that draw in a significant number of politicians, the number of (positive) mentions in mainstream ‘earned media’, the number of endorsements that are included in Congressional offices’ press releases, and the number (and relative importance) of edits made to Congressional bills.
If your work is focused on the executive or judicial branch instead of on Congress, you could adapt some of those metrics accordingly, e.g., edits to pending regulation or executive orders, or citations to your amicus curiae briefs in judicial opinions, and so on.
> frontier labs are only pretending to try to solve alignment
>>This is probably the main driver of our disagreement.
I agree with your diagnosis! I think Sam Altman is a sociopathic liar, so the fact that he signed the statement on AI risk doesn’t convince me that he cares about alignment. I feel reasonably confident about that belief. Zvi’s series on Moral Mazes apply here: I don’t claim that you literally can’t mention existential risk at OpenAI, but if you show signs of being earnestly concerned enough about it to interefere with corporate goals, then I believe you’ll be sidelined.
I’m much less confident about whether or not successful alignment looks like normal deep learning work; I know more about corporate behavior than I do about technical AI safety. It seems odd and unlikely to me that the same kind of work (normal deep learning) that looks like it causes a series of major problems (power-seeking, black boxes, emergent goals) when you do a moderate amount of it would wind up solving all of those same problems when you do a lot of it, but I’m not enough of a technical expert to be sure that that’s wrong.
Because there are independent, non-technical reasons for people to want to believe that normal deep learning will solve alignment (it means they get to take fun, high-pay, high-status jobs at AI developers without feeling guilty about it), if you show me a random person who believes this and I don’t know anything about their incorruptiability or the clarity of their thinking ahead of time, then my prior is that most of the people in the random distribution that this person was drawn from probably arrived at the belief mostly out of convenience and temptation, rather than mostly by becoming technically convinced of the merits of a position that seems a priori unlikely to me. However, I can’t be sure—perhaps it’s more likely than I think that normal deep learning can solve alignment.
Well, I can’t change the headline; I’m just a commenter. However, I think the reason why “frontier labs will fail at alignment while nonprofits can succeed” is that frontier labs are only pretending to try to solve alignment—it’s not actually a serious goal of their leadership, and it’s not likely to get meaningful support in terms of compute, recruiting, data, or interdepartmental collaboration, and in fact the leadership will probably actively interfere with your work on a regular basis because the intermediate conclusions you’re reaching will get in the way of their profits and hurt their PR. In order to do useful superalignment research, I suspect you sometimes need to warn about or at least openly discuss the serious threats that are posed by increasingly advanced AI, but the business model of frontier labs depends on pretending that none of those threats are actually serious. By contrast, the main obstacle at a nonprofit is that they might not have much funding, but at least whatever funding they do have will be earnestly directed at supporting your team’s work.
I suspect Joe would agree with me that the current odds that AI developers solve superalignment are significantly less than 20%.
Even if we concede your estimate of 20% for the sake of argument, though, what price are you likely to pay for increasing the odds of success by 0.01%? Suppose that, given enough time, nonprofit alignment researchers would eventually solve superalignment with 80% odds. In order to increase, e.g., Anthropic’s odds of success by 0.01%, are you boosting Anthropic’s capabilities in a way that shortens timelines in a way that decreases the amount of time that the nonprofit alignment teams have to solve superalignment in a way that reduces their odds of success by at least 0.0025%? If so, you’ve done net harm. If not, why not? What about Joe’s arguments that most for-profit alignment work has at least some applicability to capabilities do you find unconvincing?
Transparency is less neglected than some other topics—check out HR 5539 (Transparent by Design Act), S 3312 (AIRIA), and HR 6881 (AI Foundation Model Transparency Act).
There’s room for a little bit more useful drafting work here, but I wouldn’t call it orphaned, exactly.
I could carry on debating the pros and cons of the EO with you, but I think my real point is that bipartisan advocacy is harmless. You shouldn’t worry that bipartisan advocacy will backfire, so we can’t justify engaging in no advocacy at all out of fear that advocacy might backfire.
If you believe strongly enough in the merits of working with one party to be confident that it won’t backfire, fine, I won’t stop you—but we should all be able to agree that more bipartisan advocacy would be good, even if we disagree about how valuable one-party advocacy is.
Yes, CAIP’s strategy was primarily to get the legislation ready, talk to people about it, and prepare for a crisis. We also encouraged people to pass our legislation immediately, but I was not especially optimistic about the odds that they would agree to do so.
I don’t object to people pushing legislators from both parties to act more quickly...but you have to honor their decision if they say “no,” no matter how frustrating that is or how worried you are about the near-term future, because trying to do an end-run around their authority will quickly and predictably backfire.
In my opinion, going behind legislators’ backs to the Biden administration was particularly unhelpful for the Biden AI EO, because the contents of that EO would have led to only a small reduction in catastrophic risk—it would be nice to require reports on the results of red-teaming, but the EO by itself wouldn’t have stopped companies from reporting that their models seemed risky and then releasing them anyway. We would have needed to follow up on the EO and enact additional policies in order to have a reasonable chance of survival, but proceeding via unilateral executive action had some tendency to undermine our ability to get those additional policies passed, so it’s not clear to me what the overall theory of change was for rushing forward with an EO.
I think it’s worth distinguishing between “this AI policy could be slightly inconvenient for America’s overall geopolitical strategy” and “this AI policy is so bad for America’s AI arms race that we’re going to lose a shooting war with China.”
The former is a political problem that advocates need to find a strategy to cope with, but it’s not a reason not to do advocacy—we should be willing to trade away a little bit of American influence in order to avoid a large risk of civilizational collapse from misaligned AI.
If you’re earnestly worried about maximizing American influence, there are much better strategies for making that happen than trying to make sure that we have zero AI regulations. You could repeal the Jones Act, you could have a stable tariff regime, you could fix the visa system, you could fund either the CHIPS Act or a replacement for the CHIPS Act, you could give BIS more funding to go after chip smugglers, and so on.
I think the “concern” about the harmful geopolitical effects of moderate AI regulation is mostly opportunistic political theater by companies who would prefer to remain unregulated—there’s a notable absence of serious international relations scholars or national security experts who are coming out in favor of zero AI safety regulation as a geopolitical tool. At most, some experts might be pushing for easier land use and environmental approvals, which are not in conflict with the regulations that organizations like CAIP are pushing for.
I tend to agree with John—I think the Republican opposition is mostly a mixture of general hostility toward regulation as a tool and a desire to publicly reject the Biden Administration and all its works. My experience in talking with Republican staffers, even in 2025, is that they weren’t hostile to the idea of reducing catastrophic risks from AI; they’re just waiting to see what the Trump Administration’s official position is on the issue. We’ll know more about that when we see the OSTP AI Action Plan in July.
Part of the problem with AI safety advocacy in 2022 − 2024 was that it ignored my advice to “insist that the bills you endorse have co-sponsors from both parties.” By definition, an executive order is not bipartisan. You can frame the Biden EO as general technocracy if you like, but it’s still careless to push for an EO that’s only supported by one party. If you want to avoid alienating Republicans (and you should want to avoid doing that), then you need to make sure you have at least some Republicans willing to go on record as publicly endorsing your policy proposals before you enact them.
Hmm. There are lots of valuable advocacy targets besides literal regulation; advocates might try to pass laws, win court cases, sustain a boycott, amend a corporate charter, amend a voluntary industry standard at NIST, and so on. I’ll discuss several examples in post #5.
I suppose advocacy might also have some side benefits like teaching us more about politics, helping us bond as a community, or giving us a sense of efficacy, but it’s not obvious to me that any of those are of comparable importance to reducing the likelihood that AI developers release misaligned superintelligence.
Does that answer your question? If not, please let me know more about what you mean and I’ll try to elaborate.
I have no complaints at all about technical AI research! Indeed, I agree with you that industry will spontaneously adopt most of the good technical AI safety ideas shortly after they’re invented.
I’m arguing about the relative merits of AI governance research vs. AI governance advocacy. This probably wasn’t clear if you’re just jumping in at this point of the sequence, for which I apologize.
Thanks! If you are directly and literally comparing the utility of two or three or four concrete policy options, that’s more valuable than general academic research—it’s at least level 3, and possibly better if you write up your conclusions in a format that can be easily digested by policy wonks (level 2) or by politicians (level 1).
Unfortunately, most academic research weighs the pros and cons of general categories of policies and then offers no concrete recommendations, so a lot of what looks like “comparing different options” at first glance is not immediately valuable as a tool for picking the best currently available option.
Brainstorming new policy options to further explore the available space is at least a level 3 activity, and possibly level 2 (if you write up some of those options in enough detail that someone could advocate for them). However, as I’ll argue in the fifth post in this sequence, we already have a dozen “orphaned” policy proposals that nobody is drafting up or advocating for, so unless and until this changes, we’ll get limited value from having an even larger stack of orphaned policies. Even if you find a policy that’s an order of magnitude better than previous ideas, that policy still doesn’t generate any utility until someone goes and does the political work needed to turn it into law.
I agree that people like Bengio can be very valuable assets for AI safety advocacy, although there are diminishing marginal returns—the first computer supergenius who likes your policies is transformative; the third is helpful; the tenth is mostly just a statistic in a survey and will not meaningfully change, e.g., journalists or staffers’ opinions about an issue.
If you think that technology or history will move far enough ahead that people like Bengio and Hinton will lose their relevance, then it might be a good idea to try to convince the next Bengio to support some AI safety policies. If that’s your strategy, then you should develop a short list of people who might be the next Bengio, and then go find them and talk to them in person. Once you’ve identified some leading young computer scientists and some questions that they’re uncertain and curious about, then you can go do research to try to help convince them to take a particular stance on those questions.
Just publishing AI governance research of general academic interest is very far removed from the goal of recruiting computer science superstars for AI x-risk advocacy.
Yes, that’s exactly right, we do. That’s what it means to be an ally rather than a friend. America allied with the Soviet Union in World War 2; this is no different. When someone earnestly offers to help you literally save the world, you hold your nose and shake their hand.