I don’t think this outcome was overdetermined; there’s no recent medical breakthrough behind this progress. It just took a herculean act of international coordination and logistics. It took distributing millions of water filters, establishing village-based surveillance systems in thousands of villages across multiple countries, and meticulously tracking every single case of Guinea worm in humans or livestock around the world. It took brokering a six-month ceasefire in Sudan (the longest humanitarian ceasefire in history!) to allow healthcare workers to access the region. I’ve only skimmed the history, and I’m generally skeptical of historical heroes getting all the credit, but I tentatively think it took Jimmy Carter for all of this to happen.
I’m compelled to caveat that top GiveWell charities are probably in the ballpark of $50/DALY, and the Carter Center has an annual budget of ~$150 million a year, so they “should” be able to buy 2 million DALYs every single year by donating to more cost-effective charities. But c’mon this worm is super squicky and nearly eradicating it is an amazing act of agency.
I don’t think you need that footnoted caveat, simply because there isn’t $150M/year worth of room for more funding in all of AMF, Malaria Consortium’s SMC program, HKI’s vitamin A supplementation program, and New Incentives’ cash incentives for routine vaccination program all combined; these comprise the full list of GiveWell’s top charities.
Another point is that the benefits of eradication keep adding up long after you’ve stopped paying for the costs, because the counterfactual that people keep suffering and dying of the disease is no longer happening. That’s how smallpox eradication’s cost-effectiveness can plausibly be less than a dollar per DALY averted so far and dropping (Guesstimate model, analysis). Quoting that analysis:
3.10.) For how many years should you consider benefits?
It is not clear for how long we should continue to consider benefits, since the benefits of vaccines would potentially continue indefinitely for hundreds of years. Perhaps these benefits would eventually be offset by some other future technology, and we could try to model that. Or perhaps we should consider a discount rate into the future, though we don’t find that idea appealing.
Instead, we decided to cap at an arbitrary fixed amount of years set to 20 by default, though adjustable as a variable in our spreadsheet model (or by copying and modifying our Guesstimate models). We picked 20 because it felt like a significant enough amount of time for technology and other dynamics to shift.
It’s important to think through what cap makes the most sense, though, as it can have a large effect on the final model, as seen in this table where we explore the ramifications of smallpox eradication with different benefit thresholds:
Here’s my attempt at a neutral look at Prop 50, which people in California can vote on Tuesday (Nov 4th). The bill seems like a case-study in high-stakes game theory and when to cooperate or defect.
The bill would allow the CA legislature to re-write the congressional district maps until 2030 (when district-drawing would go back to normal). Currently, the district maps are drawn by an independent body designed to be politically neutral. In essence, this would allow the CA legislature to gerrymander California. That would probably give Democrats an extra 3-5 seats in Congress. It seems like there’s a ~17% chance that it swings the House in the midterms.
Gerrymandering is generally agreed to be a bad thing, since it means elections are determined on the margin more by the map makers and less by the people. The proponents of this bill don’t seem to think otherwise. They argue the bill is in response to Texas passing a similar bill to redistrict in a way that is predicted to give Republicans 5 new house seats (not to mention similar bills in North Carolina and Missouri that would give republicans an additional 2 seats).
Trump specifically urged Texas, North Carolina, and Missouri to pass their bills, and the rationale was straightforwardly to give Republicans a greater chance at winning the midterms. For example, Rep. Todd Hunter, the author of Texas’s redistricting bill, said “The underlying goal of this plan is straightforward, [to] improve Republican political performance”.
Notably some Republicans have also tried to argue that the Texas bill is in response to Democrats gerrymandering and obstructionism, but this doesn’t match how Trump seems to have described the rationale originally.[1]
The opponents of Prop 50 don’t seem to challenge the notion that the Republican redistricting was bad.[2] They just argue that gerrymandering is bad for all the standard reasons.
So, it’s an iterated prisoners’ dilemma! Gerrymandering is bad, but the Republicans did it, maybe the Democrats should do it to (1) preserve political balance and (2) punish/disincentivize Republicans’ uncooperative behavior.
Some questions you might have:
Will this actually disincentivize gerrymandering? Maybe the better way to disincentivize it is to set a good example.
Generally I’m skeptical of arguments like “the other guys defect in this prisoners’ dilemma and so you should too”. In practice, it’s often hard to tell why someone is defecting or for the counterparty to credibly signal that they would in fact switch to the cooperate-cooperate equilibrium if it was available. Real life is messy, it’s easy to defect and blame it on your counterparty defecting even when they didn’t, and being the kind of person who will legibly reliably cooperate when it counts is very valuable. For these reasons I tend to err towards being cooperative in practice.
In this case, if CA passes Prop 50, maybe republican voters won’t see it as a consequence of Republican gerrymandering and will simply interpret it as “the Democrats gerrymander and go whatever uncooperative behavior gets them the most votes. We need to do whatever it takes to win” or “everyone gerrymanders, gerrymandering is normal and just part and parcel of how the sausage is made”.
On top of that, I’m wary of ending up in one of the defect-defect equilibria tit-for-tat is famous for. Tit-for-two-tats and forgiveness are sometimes helpful.
But I think Prop 50 handles these things fairly well. The bill only lasts until 2030 and has been framed explicitly and clearly as in direct response to redistricting in Texas. (In fact Governor Newsom’s original proposal was to make Prop 50 “Preserves California’s current congressional maps if Texas or other states also keep their original maps.” That provision was removed once Texas solidified its redistricting.) Fretting too much about if Republicans will take even more aggressive actions because of this bill also incentives Republicans to be more aggressive in their responses and to pay less attention to Democrats’ rationales, which seems bad.
Moreover, if Democrats are benefiting similarly to Republicans from gerrymandering, perhaps this creates more bipartisan support for federal regulation banning gerrymandering. In general, where possible, I think it’s good to have laws preventing this kind of uncooperative behavior rather than relying on both parties managing to hit cooperate in a complicated prisoner’s dilemma.
Are the costs to society simply too large to be worth it?
In some ways, Prop 50 undoes some of the damage of redistricting in Texas: in Texas republicans gained 5 seats in a way that isn’t as representative as it should have been, so by undoing that and giving Democrats 3-5 extra seats, the system becomes more representative. But in some ways two wrongs don’t make a right here: at the end of the day both Texans and California end up less representative. For instance, if you think it’s more important for congress being made up of politicians who represent their constituents well and less important that constituents’ views are represented federally.
Notably even if you buy that argument you might still think Prop 50 is worth it if you think the punishing effects are worth it.
What’s the historical context? If this is a prisoner’s dilemma, how much has each side hit cooperate in the past?
Republicans have sometimes said their redistricting bills are a response to Democrats’ gerrymandering. If so, maybe they’re justified. Let’s look into it! You can read the history here or look at an interactive map here.
It seems like Republicans engaged in a major, unprovoked bout of gerrymandering in 2010 with REDMAP. Since then both parties have tried to gerrymander and occasionally succeeded. Overall, Republicans have gerrymandered somewhat more than Democrats, but Democrats have still engaged in blatant gerrymandering, for example, in Illinois in 2021. In searching for more right-leaning narratives, I found that Brookings estimated in 2023 that no party majorly benefited from gerrymandering more than another at the time, regardless of how much they’d engaged in it. I haven’t really found a great source for anyone claiming Democrats have overall benefited more from gerrymandering.
Democrats have also tried to propose a bill to ban gerrymandering federally, the Freedom to Vote Act. (This bill also included some other provisions apart from just banning gerrymandering, like expanding voter registration and making Election Day a federal holiday.) The Freedom to Vote Act was widely opposed by Republicans and I don’t know of any similar legislation they’ve proposed to ban gerrymandering.
So overall, it seems like Republicans have been engaging in more gerrymandering than Democrats and been doing less to fix the issue.
Republicans have also argued the new districts in Texas represent the Hispanic population better, though they tend to frame this more as a reason it’s good and less as the reason they pursued this redistricting in the first place.
Specifically, they say “While Newsom and CA Democrats say Prop 50 is a response to Trump and Texas redistricting, California shouldn’t retaliate and sacrifice its integrity by ending fair elections.”
One argument against the bill that I didn’t explore above (because I haven’t actually heard anyone make it) is that the only reason Democrats aren’t gerrymandering more is because gerrymandering seems more helpful to Republicans for demographic reasons. But Democrats try to do other things that are arguably designed to give them more votes. For example, loosening voter ID laws. So maybe each party should carefully respond to the ways the other party tries to sneakily get themselves more votes in very measured ways that properly disincentivize bad behavior engage in a crazy ever-escalating no-holds-barred race to the bottom.
I think it’s good that the Republicans and Democrats have been somewhat specific that their attempts at gerrymandering are only retaliation against other gerrymandering, and not retaliation against things like this
To elaborate on this, a model of voting demographics is that the most engaged voters vote no matter what hoops they need to jump through, so rules and laws that make voting easier increase the share of less engaged voters. This benefits whichever party is comparatively favored by these less engaged voters. Historically this used to be the Democrats, but due to education polarization they’ve become the party of the college-educated nowadays. This is also reflected in things like Trump winning the Presidential popular vote in 2024. (Though as a counterpoint, this Matt Yglesias article from 2022 claims that voter ID laws “do not have a discernible impact on election results” but doesn’t elaborate.)
In addition, voter ID laws are net popular, so Democrats advocating against them hurts them both directly (advocating for an unpopular policy) and indirectly (insofar as it increases the pool of less engaged voters).
Seen in the light of Section 2 of the Voting Rights Act asymmetrically binding Republicans, what you’re calling an “unprovoked bout of gerrymandering” might be better understood as an attempt to reduce the unfair advantage Democrats have had nationally for decades.
The problem with gerrymandering is that it makes elections less representative. It seems to me that (section 2 of) the Voting Rights Act makes elections more representative, so that’s good. It seems reasonable to be mad at republicans when they implement measures that make elections less representative that benefit them, but not when you want elections to stay less fair.
Ideas for how to spend very large amounts of money to improve AI safety:
If AI companies’ valuations continue to skyrocket (or if new very wealthy actors start to become worried about AI risk), there might be a large influx of funding into the AI safety space. Unfortunately, it’s not straightforward to magically turn money into valuable AI safety work. Many things in the AI safety ecosystem are more bottlenecked on having a good founder with the right talent and context, or having good researchers.
Here’s a random incomplete grab-bag of ideas for ways you could turn money into reductions in AI risk at large scales. I think right now there are much better donation opportunities available. This is not a list of donation recommendations right now, it’s just suggestions for once all the low-hanging funding fruit has been plucked. Probably if people thought more they could come up with even better scalable opportunities. There’s also probably existing great ideas I neglected to list. But these at least give us a baseline and a rough sense of what dumping a bunch of money into AI safety could look like. I’m also erring towards listing more things rather than fewer. Some of these things might actually be bad ideas.
Bounties to reward AIs for reporting misaligned behavior in themselves or other agents.
Folks have run a couple small experiments on this already. It seems straight-forward to execute and like it could absorb almost unbounded amounts of capital.
Paying high enough salaries to entice non-altruistically-motivated AI company employees to work on safety.
This isn’t only bottlenecked on funding. Many people are very loyal to the AI companies they work for, and the very best employees aren’t very sensitive to money since they already have plenty of money. It seems absurdly expensive for Meta to try hiring away people at other AI companies, and they didn’t seem to get that much top talent from it. On the one hand, working on safety is a much more compelling case than working at Meta, but on the other hand, maybe people who aren’t already doing safety research find AI capabilities research more intrinsically fun and interesting or rewarding than safety research. I am also concerned that people who do capabilities research might not be great at safety research because they might not feel as passionate or inspired by it, and because it is a somewhat different skillset.
In the most extremely optimistic world, you could probably hire 50 extremely talented people by offering them $100M/year each (matching what Meta offered). You could probably also hire ~200 more junior people at $10M/year (the bottleneck on hiring more would be management capacity). So in total you could spend $7B/year.
Over time, I expect this to get more expensive since AI companies’ valuations will increase, and therefore, so will employee compensation.
Compute for AI safety research.
Day-to-day, the AI safety researchers I know outside of AI labs don’t seem to think they’re very bottlenecked on compute. However, the AI safety researchers I know inside AI labs claim they get a lot of value from having gobs and gobs of compute everywhere. Probably, AI safety researchers outside labs are just not being imaginative enough about what they could do with tons of compute. This also isn’t entirely money-bottlenecked. Probably part of it is having the infrastructure in place and the deals with the compute providers, etc. And running experiments on lots of compute can be more fiddly and time-consuming. Even so I bet with a lot more money for compute, people would be able to do much better safety research.
Very roughly, I guess this could absorb ~$100 million a year.
Compute for running AI agents to automate AI safety research.
This doesn’t work today since AIs can’t automate AI safety research. But maybe in the future they will be able to, and you’ll be able to just dump money into this almost indefinitely.
Pay AI companies to do marginal cheap safety interventions.
Maybe you can just pay AI companies to implement safety interventions that are only very slightly costly for them. For example, you could subsidize having really good physical security in their data centers. I think a lot of things AI companies could do to improve safety will be costly enough for the companies that it will be very hard to pay them enough to make up for that cost, especially in worlds where AI companies’ valuations have increased a lot from where they are today. But there’s probably still some opportunities here.
Raising awareness of AI safety.
There’s lots of proven ways to spend money to raise awareness of things (sponsor youtube channels, patronize movies about AI risk, etc). Maybe raising awareness of safety is good because it gets more people to work on safety or gets the government to do more sensible things about AI risk or lets consumers encourage companies to implement more safety interventions.
I couldn’t easily find an American public awareness campaign that cost more than ~$80M/year (for anti-smoking). Coca Cola spends ~$4 billion a year on advertising, but I think that if AI safety were spending as much money as Coca-Cola, it would backfire. I think maybe $500M/year is a reasonable cap on what could be spent?
Biodefense. Buy everyone in the US PPE.
One way that an AI could cause a catastrophe is via designing a bioweapon. One way to reduce the odds that a bioweapon causes a civilization-ending catastrophe is to make sure that everyone has enough PPE that they won’t die. Andrew Snyder-Beattie has elaborated on this idea here. I think this could absorb ~$3B ($3/mask * 350M Americans * 3 masks/person).
Buy foreign AI safety researchers gold cards.
Many great AI safety researchers are on visas. It would be convenient if they had green cards. You can buy green cards now for $1M each. Let’s say there’s a hundred such people, so this opportunity could absorb $100M.
Overall, these are not amazing opportunities. But they give a lower bound and illustrate how it’s possible to turn money into reduced risk from AI at scale, even if you don’t have more entrepreneurs building new organizations. In practice, I think if money slowly ramps up into the space over time, there will be much better opportunities than these, and you will simply see AI safety organizations that have grown to be major research institutions that are producing wonderful research. This is just a floor.
A lot of these ideas came from other people and have generally been floating around for a while. Thanks to everybody I talk to about this.
In the most extremely optimistic world, you could probably hire 50 extremely talented people by offering them $100M/year each (matching what Meta offered). You could probably also hire ~200 more junior people at $10M/year (the bottleneck on hiring more would be management capacity). So in total you could spend $7B/year.
Over time, I expect this to get more expensive since AI companies’ valuations will increase, and therefore, so will employee compensation.
I don’t know that the idea is fundamentally good but at least is scales somewhat with the equity of the safety-sympathetic people at labs?
Humanity has only ever eradicated two diseases (and one of those, rinderpest, is only in cattle not humans). The next disease on the list is probably Guinea worm (though polio is also tantalizingly close).
At its peak Guinea worm infected ~900k people a year. In 2024 we so far only know of 7 cases. The disease isn’t deadly, but it causes significant pain for 1-3 weeks (as a worm burrows out of your skin!) and in ~30% of cases that pain persists afterwards for about a year. In .5% of cases the worm burrows through important ligaments and leaves you permanently disabled. Eradication efforts have already saved about 2 million DALYs.[1]
I don’t think this outcome was overdetermined; there’s no recent medical breakthrough behind this progress. It just took a herculean act of international coordination and logistics. It took distributing millions of water filters, establishing village-based surveillance systems in thousands of villages across multiple countries, and meticulously tracking every single case of Guinea worm in humans or livestock around the world. It took brokering a six-month ceasefire in Sudan (the longest humanitarian ceasefire in history!) to allow healthcare workers to access the region. I’ve only skimmed the history, and I’m generally skeptical of historical heroes getting all the credit, but I tentatively think it took Jimmy Carter for all of this to happen.
Rest in peace, Jimmy Carter.
I’m compelled to caveat that top GiveWell charities are probably in the ballpark of $50/DALY, and the Carter Center has an annual budget of ~$150 million a year, so they “should” be able to buy 2 million DALYs every single year by donating to more cost-effective charities. But c’mon this worm is super squicky and nearly eradicating it is an amazing act of agency.
I don’t think you need that footnoted caveat, simply because there isn’t $150M/year worth of room for more funding in all of AMF, Malaria Consortium’s SMC program, HKI’s vitamin A supplementation program, and New Incentives’ cash incentives for routine vaccination program all combined; these comprise the full list of GiveWell’s top charities.
Another point is that the benefits of eradication keep adding up long after you’ve stopped paying for the costs, because the counterfactual that people keep suffering and dying of the disease is no longer happening. That’s how smallpox eradication’s cost-effectiveness can plausibly be less than a dollar per DALY averted so far and dropping (Guesstimate model, analysis). Quoting that analysis:
Here’s my attempt at a neutral look at Prop 50, which people in California can vote on Tuesday (Nov 4th). The bill seems like a case-study in high-stakes game theory and when to cooperate or defect.
The bill would allow the CA legislature to re-write the congressional district maps until 2030 (when district-drawing would go back to normal). Currently, the district maps are drawn by an independent body designed to be politically neutral. In essence, this would allow the CA legislature to gerrymander California. That would probably give Democrats an extra 3-5 seats in Congress. It seems like there’s a ~17% chance that it swings the House in the midterms.
Gerrymandering is generally agreed to be a bad thing, since it means elections are determined on the margin more by the map makers and less by the people. The proponents of this bill don’t seem to think otherwise. They argue the bill is in response to Texas passing a similar bill to redistrict in a way that is predicted to give Republicans 5 new house seats (not to mention similar bills in North Carolina and Missouri that would give republicans an additional 2 seats).
Trump specifically urged Texas, North Carolina, and Missouri to pass their bills, and the rationale was straightforwardly to give Republicans a greater chance at winning the midterms. For example, Rep. Todd Hunter, the author of Texas’s redistricting bill, said “The underlying goal of this plan is straightforward, [to] improve Republican political performance”.
Notably some Republicans have also tried to argue that the Texas bill is in response to Democrats gerrymandering and obstructionism, but this doesn’t match how Trump seems to have described the rationale originally.[1]
The opponents of Prop 50 don’t seem to challenge the notion that the Republican redistricting was bad.[2] They just argue that gerrymandering is bad for all the standard reasons.
So, it’s an iterated prisoners’ dilemma! Gerrymandering is bad, but the Republicans did it, maybe the Democrats should do it to (1) preserve political balance and (2) punish/disincentivize Republicans’ uncooperative behavior.
Some questions you might have:
Will this actually disincentivize gerrymandering? Maybe the better way to disincentivize it is to set a good example.
Generally I’m skeptical of arguments like “the other guys defect in this prisoners’ dilemma and so you should too”. In practice, it’s often hard to tell why someone is defecting or for the counterparty to credibly signal that they would in fact switch to the cooperate-cooperate equilibrium if it was available. Real life is messy, it’s easy to defect and blame it on your counterparty defecting even when they didn’t, and being the kind of person who will legibly reliably cooperate when it counts is very valuable. For these reasons I tend to err towards being cooperative in practice.
In this case, if CA passes Prop 50, maybe republican voters won’t see it as a consequence of Republican gerrymandering and will simply interpret it as “the Democrats gerrymander and go whatever uncooperative behavior gets them the most votes. We need to do whatever it takes to win” or “everyone gerrymanders, gerrymandering is normal and just part and parcel of how the sausage is made”.
On top of that, I’m wary of ending up in one of the defect-defect equilibria tit-for-tat is famous for. Tit-for-two-tats and forgiveness are sometimes helpful.
But I think Prop 50 handles these things fairly well. The bill only lasts until 2030 and has been framed explicitly and clearly as in direct response to redistricting in Texas. (In fact Governor Newsom’s original proposal was to make Prop 50 “Preserves California’s current congressional maps if Texas or other states also keep their original maps.” That provision was removed once Texas solidified its redistricting.) Fretting too much about if Republicans will take even more aggressive actions because of this bill also incentives Republicans to be more aggressive in their responses and to pay less attention to Democrats’ rationales, which seems bad.
Moreover, if Democrats are benefiting similarly to Republicans from gerrymandering, perhaps this creates more bipartisan support for federal regulation banning gerrymandering. In general, where possible, I think it’s good to have laws preventing this kind of uncooperative behavior rather than relying on both parties managing to hit cooperate in a complicated prisoner’s dilemma.
Are the costs to society simply too large to be worth it?
In some ways, Prop 50 undoes some of the damage of redistricting in Texas: in Texas republicans gained 5 seats in a way that isn’t as representative as it should have been, so by undoing that and giving Democrats 3-5 extra seats, the system becomes more representative. But in some ways two wrongs don’t make a right here: at the end of the day both Texans and California end up less representative. For instance, if you think it’s more important for congress being made up of politicians who represent their constituents well and less important that constituents’ views are represented federally.
Notably even if you buy that argument you might still think Prop 50 is worth it if you think the punishing effects are worth it.
What’s the historical context? If this is a prisoner’s dilemma, how much has each side hit cooperate in the past?
Republicans have sometimes said their redistricting bills are a response to Democrats’ gerrymandering. If so, maybe they’re justified. Let’s look into it! You can read the history here or look at an interactive map here.
It seems like Republicans engaged in a major, unprovoked bout of gerrymandering in 2010 with REDMAP. Since then both parties have tried to gerrymander and occasionally succeeded. Overall, Republicans have gerrymandered somewhat more than Democrats, but Democrats have still engaged in blatant gerrymandering, for example, in Illinois in 2021. In searching for more right-leaning narratives, I found that Brookings estimated in 2023 that no party majorly benefited from gerrymandering more than another at the time, regardless of how much they’d engaged in it. I haven’t really found a great source for anyone claiming Democrats have overall benefited more from gerrymandering.
Democrats have also tried to propose a bill to ban gerrymandering federally, the Freedom to Vote Act. (This bill also included some other provisions apart from just banning gerrymandering, like expanding voter registration and making Election Day a federal holiday.) The Freedom to Vote Act was widely opposed by Republicans and I don’t know of any similar legislation they’ve proposed to ban gerrymandering.
So overall, it seems like Republicans have been engaging in more gerrymandering than Democrats and been doing less to fix the issue.
Republicans have also argued the new districts in Texas represent the Hispanic population better, though they tend to frame this more as a reason it’s good and less as the reason they pursued this redistricting in the first place.
Specifically, they say “While Newsom and CA Democrats say Prop 50 is a response to Trump and Texas redistricting, California shouldn’t retaliate and sacrifice its integrity by ending fair elections.”
One argument against the bill that I didn’t explore above (because I haven’t actually heard anyone make it) is that the only reason Democrats aren’t gerrymandering more is because gerrymandering seems more helpful to Republicans for demographic reasons. But Democrats try to do other things that are arguably designed to give them more votes. For example, loosening voter ID laws. So maybe each party should
carefully respond to the ways the other party tries to sneakily get themselves more votes in very measured ways that properly disincentivize bad behaviorengage in a crazy ever-escalating no-holds-barred race to the bottom.I think it’s good that the Republicans and Democrats have been somewhat specific that their attempts at gerrymandering are only retaliation against other gerrymandering, and not retaliation against things like this
My understanding is that voter ID laws are probably net helpful for Democrats at this point.
To elaborate on this, a model of voting demographics is that the most engaged voters vote no matter what hoops they need to jump through, so rules and laws that make voting easier increase the share of less engaged voters. This benefits whichever party is comparatively favored by these less engaged voters. Historically this used to be the Democrats, but due to education polarization they’ve become the party of the college-educated nowadays. This is also reflected in things like Trump winning the Presidential popular vote in 2024. (Though as a counterpoint, this Matt Yglesias article from 2022 claims that voter ID laws “do not have a discernible impact on election results” but doesn’t elaborate.)
In addition, voter ID laws are net popular, so Democrats advocating against them hurts them both directly (advocating for an unpopular policy) and indirectly (insofar as it increases the pool of less engaged voters).
Seen in the light of Section 2 of the Voting Rights Act asymmetrically binding Republicans, what you’re calling an “unprovoked bout of gerrymandering” might be better understood as an attempt to reduce the unfair advantage Democrats have had nationally for decades.
The problem with gerrymandering is that it makes elections less representative. It seems to me that (section 2 of) the Voting Rights Act makes elections more representative, so that’s good. It seems reasonable to be mad at republicans when they implement measures that make elections less representative that benefit them, but not when you want elections to stay less fair.
Ideas for how to spend very large amounts of money to improve AI safety:
If AI companies’ valuations continue to skyrocket (or if new very wealthy actors start to become worried about AI risk), there might be a large influx of funding into the AI safety space. Unfortunately, it’s not straightforward to magically turn money into valuable AI safety work. Many things in the AI safety ecosystem are more bottlenecked on having a good founder with the right talent and context, or having good researchers.
Here’s a random incomplete grab-bag of ideas for ways you could turn money into reductions in AI risk at large scales. I think right now there are much better donation opportunities available. This is not a list of donation recommendations right now, it’s just suggestions for once all the low-hanging funding fruit has been plucked. Probably if people thought more they could come up with even better scalable opportunities. There’s also probably existing great ideas I neglected to list. But these at least give us a baseline and a rough sense of what dumping a bunch of money into AI safety could look like. I’m also erring towards listing more things rather than fewer. Some of these things might actually be bad ideas.
Bounties to reward AIs for reporting misaligned behavior in themselves or other agents.
Folks have run a couple small experiments on this already. It seems straight-forward to execute and like it could absorb almost unbounded amounts of capital.
Paying high enough salaries to entice non-altruistically-motivated AI company employees to work on safety.
This isn’t only bottlenecked on funding. Many people are very loyal to the AI companies they work for, and the very best employees aren’t very sensitive to money since they already have plenty of money. It seems absurdly expensive for Meta to try hiring away people at other AI companies, and they didn’t seem to get that much top talent from it. On the one hand, working on safety is a much more compelling case than working at Meta, but on the other hand, maybe people who aren’t already doing safety research find AI capabilities research more intrinsically fun and interesting or rewarding than safety research. I am also concerned that people who do capabilities research might not be great at safety research because they might not feel as passionate or inspired by it, and because it is a somewhat different skillset.
In the most extremely optimistic world, you could probably hire 50 extremely talented people by offering them $100M/year each (matching what Meta offered). You could probably also hire ~200 more junior people at $10M/year (the bottleneck on hiring more would be management capacity). So in total you could spend $7B/year.
Over time, I expect this to get more expensive since AI companies’ valuations will increase, and therefore, so will employee compensation.
Compute for AI safety research.
Day-to-day, the AI safety researchers I know outside of AI labs don’t seem to think they’re very bottlenecked on compute. However, the AI safety researchers I know inside AI labs claim they get a lot of value from having gobs and gobs of compute everywhere. Probably, AI safety researchers outside labs are just not being imaginative enough about what they could do with tons of compute. This also isn’t entirely money-bottlenecked. Probably part of it is having the infrastructure in place and the deals with the compute providers, etc. And running experiments on lots of compute can be more fiddly and time-consuming. Even so I bet with a lot more money for compute, people would be able to do much better safety research.
Very roughly, I guess this could absorb ~$100 million a year.
Compute for running AI agents to automate AI safety research.
This doesn’t work today since AIs can’t automate AI safety research. But maybe in the future they will be able to, and you’ll be able to just dump money into this almost indefinitely.
Pay AI companies to do marginal cheap safety interventions.
Maybe you can just pay AI companies to implement safety interventions that are only very slightly costly for them. For example, you could subsidize having really good physical security in their data centers. I think a lot of things AI companies could do to improve safety will be costly enough for the companies that it will be very hard to pay them enough to make up for that cost, especially in worlds where AI companies’ valuations have increased a lot from where they are today. But there’s probably still some opportunities here.
Raising awareness of AI safety.
There’s lots of proven ways to spend money to raise awareness of things (sponsor youtube channels, patronize movies about AI risk, etc). Maybe raising awareness of safety is good because it gets more people to work on safety or gets the government to do more sensible things about AI risk or lets consumers encourage companies to implement more safety interventions.
I couldn’t easily find an American public awareness campaign that cost more than ~$80M/year (for anti-smoking). Coca Cola spends ~$4 billion a year on advertising, but I think that if AI safety were spending as much money as Coca-Cola, it would backfire. I think maybe $500M/year is a reasonable cap on what could be spent?
Biodefense. Buy everyone in the US PPE.
One way that an AI could cause a catastrophe is via designing a bioweapon. One way to reduce the odds that a bioweapon causes a civilization-ending catastrophe is to make sure that everyone has enough PPE that they won’t die. Andrew Snyder-Beattie has elaborated on this idea here. I think this could absorb ~$3B ($3/mask * 350M Americans * 3 masks/person).
Buy foreign AI safety researchers gold cards.
Many great AI safety researchers are on visas. It would be convenient if they had green cards. You can buy green cards now for $1M each. Let’s say there’s a hundred such people, so this opportunity could absorb $100M.
Overall, these are not amazing opportunities. But they give a lower bound and illustrate how it’s possible to turn money into reduced risk from AI at scale, even if you don’t have more entrepreneurs building new organizations. In practice, I think if money slowly ramps up into the space over time, there will be much better opportunities than these, and you will simply see AI safety organizations that have grown to be major research institutions that are producing wonderful research. This is just a floor.
A lot of these ideas came from other people and have generally been floating around for a while. Thanks to everybody I talk to about this.
I don’t know that the idea is fundamentally good but at least is scales somewhat with the equity of the safety-sympathetic people at labs?