Coefficient Giving (where I work as a grantmaker) recently launched Project Tailwind, a call for founders of AI safety orgs[1]. Many of the best existing AI safety orgs were started by LessWrong readers, and I expect some of the best orgs we fund through Tailwind will be too.
The website includes descriptions of 36 potential orgs that various CG grantmakers would like to fund, which I’ve pasted in a list below. But these are just examples – you can apply for funding to start[2] any org that has a good shot at reducing catastrophic risk from AI. The strategic situation has been changing pretty fast, so there’s probably plenty of great org ideas that aren’t on this list.
So funding isn’t really a bottleneck. But talent is! So if you think one of these orgs is a promising way to reduce x-risk, maybe you should be the person to make it happen. And if you think they’re all rubbish ideas, maybe you should apply with a better one. And if you don’t want to start an org, maybe you should post your better idea in the comments for someone else to use. It’s also valuable to post critiques of these ideas.
We’re using a loose framework of pre-seed grants of $200k–$2m, seed grants of $2m–$20m, and bigger bets of $20m–$200m or even more. I expect that we’ll often give out pre-seed grants to promising people with good ideas, even when their plans are not particularly fleshed out and they expect to pivot around a lot. We also fund generous compute budgets and competitive salaries.
The easiest way to apply for funding is to fill out this quick expression-of-interest form. If your EOI sounds promising we’ll invite you to submit a full proposal.
Full list of example orgs:
Independent assessment
Frontier lab accountability: Independent tracking, comparison, and review of frontier labs’ safety-relevant practices.
Misalignment red team: Discover misalignment properties in frontier models before the world finds out the hard way.
Control red team: Stress-test AI companies’ security and control protocols before a misaligned AI does.
Independent AI auditors: Build the independent audit capacity necessary for AI regulation to succeed.
Investigations for secret loyalties: Determine whether companies can prevent a small group of people from bending AI to their own interests.
Misleadingness evaluations: Build evaluations that incentivize labs to make models less misleading on fuzzy tasks.
Research institutions
Oversight of CoT monitorability: Measure how monitorable frontier model chains of thought are, track if that property degrades, and identify ways to preserve it.
AI character evaluator: Improve our understanding of AIs’ behavioral tendencies and overall values.
Unclaimed research labs: Build a research lab around one of the many promising safety agendas still waiting for a team.
Training run fault injections: Construct misaligned models in vitro to map which training procedures contribute to misalignment.
Threat modeling institute: Develop and curate the best models of how catastrophic risks from AI could occur.
Alignment moonshots: Pioneer alignment techniques robust enough to trust with superintelligent AI.
Just for common knowledge, while Dustin is no longer blocking funding to rationalist-adjacent organizations, CG has still rejected funding Lightcone Infrastructure (despite Dustin actively wanting to make a grant to us[1]). I find it frustrating to say that “funding isn’t really a bottleneck” while you continue to block funding to a lot of the historically most effective organizations (by the lights of your own staff!).
Funding continues to be a non-trivial bottleneck for many of the best organizations. I continue to be highly concerned about the pressures and selection effects that apply when an organization is trying to get funding from CG.
Which he indeed ended up doing directly from Good Ventures after (as far as I know) CG-leadership vetoed grant recommendations by its staff to us (and which I am very grateful to Dustin for doing).
I find it frustrating to say that “funding isn’t really a bottleneck” while you continue to block funding to a lot of the historically most effective organizations (by the lights of your own staff!)
Fair enough, seems true that funding is still a bottleneck for orgs that CG doesn’t fund, and maybe it’s clumsy/insensitive to throw that phrase around given that.
There are, at least, many impactful things one can do that are not currently bottlenecked on CG’s willingness to fund them, but on the availability of competent people to do them.
(Unsure how helpful pointing this out is given that CG has probably gotten this critique before, but pointing it out anyways on the off chance it is helpful.)
“The range of AI safety work that CG is willing to fund is pretty broad these days, too.”
On the contrary it seems like your list of examples has a pretty narrow range. Much of it seems to be underpinned by a worldview that is something like “LLMs scale smoothly and predictably to AGI within the frontier labs and within a few years to a decade” (with some aspects of this being more or less load-bearing than others). I get that this view looks quite favorable right now, and I get that it’s hard to find ideas for deploying a large amount of funding under alternatives to this view, but I’d love to see more effort put into seeking out and funding weird alternative research programs and projects that help the field be diversified enough to be robust to some aspects of this worldview being wrong. Even if this view is entirely correct, it still seems to me that these projects are too highly correlated with each other and more decorrelated bets can be made.
Funding Tsvi’s HIA project would be a good start.
If CG in fact agrees with this idea, then signaling this more clearly would be ideal.
Just to share my thoughts after seeing the announcement and how I’ve been thinking transitioning from my previous work (in case it’s helpful to others in a similar boat):
I intend to apply, though I expect it will take me ~2-3 months to prepare the application (not the actual writing of it). In my case, my guess is that I will need to dedicate time to find co-founders and advisors before my application will be taken seriously (which is how I see what “funding is not a bottleneck, but talent is” is pointing to). Multiple people have told me that what I’m pursuing fits exactly into what CG and other funders want, but my guess is that I need to form a strong team first (perhaps with those who have gained more status in the field) and some preliminary related work.
As a result, I’m hoping to get short-term funding so I can even have the runway to apply properly. I was hoping to apply for much more funding faster so that I could then have the funding and take my time work trialing co-founders, but I hopefully I can get short-term funding to fill the gap and I don’t remain unpaid for months. I am mostly anxious about spending months preparing for this while not being paid and draining all my funds in the last ~1.5 years pursuing an AI safety startup (because I was hoping to bring in more money to AI safety since there wasn’t much philanthropic spending), living very frugally.
You might be overestimating how developed a project has to be to get pre-seed funding. I can think of several recent grants we’ve made to very early-stage projects. So I think it’s probably worth submitting an EOI now
> In my case, my guess is that I will need to dedicate time to find co-founders and advisors before my application will be taken seriously (which is how I see what “funding is not a bottleneck, but talent is” is pointing to).
Good co-founders and advisors can be help in getting funding, but by “talent is a bottleneck” I just meant “we could really do with more great people eager to start projects”.
I’ll consider applying sooner. My guess has been that the bar for talent will be high, and asking for funding anywhere near what I’d like would become a bottleneck because I don’t yet have enough ‘talent’. I’ve been assuming I can ask for much more in ~3 months, after I have a strong team (vs. telling funders, “I’m actively looking for co-founders who will fill roles x and y”). Maybe the happy-medium solution is to ask for less money ASAP, then apply for more funding every few months as we seek to grow.
What kind of submission timeline is helpful for a pre-seed EOI? Is it something where it’s beneficial to have it in within the next few days or is it okay to take a few weeks to solidify the proposal more?
Coefficient Giving (where I work as a grantmaker) recently launched Project Tailwind, a call for founders of AI safety orgs[1]. Many of the best existing AI safety orgs were started by LessWrong readers, and I expect some of the best orgs we fund through Tailwind will be too.
The website includes descriptions of 36 potential orgs that various CG grantmakers would like to fund, which I’ve pasted in a list below. But these are just examples – you can apply for funding to start[2] any org that has a good shot at reducing catastrophic risk from AI. The strategic situation has been changing pretty fast, so there’s probably plenty of great org ideas that aren’t on this list.
CG now moves pretty fast and goes pretty big compared to historical baselines. For example, in June my team got approval for up to $160m to found Resolution in about 6 weeks. And in August we got a $10m grant to Jacob Tsimerman approved 12 days after he first reached out to us. The range of AI safety work that CG is willing to fund is pretty broad these days, too.
So funding isn’t really a bottleneck. But talent is! So if you think one of these orgs is a promising way to reduce x-risk, maybe you should be the person to make it happen. And if you think they’re all rubbish ideas, maybe you should apply with a better one. And if you don’t want to start an org, maybe you should post your better idea in the comments for someone else to use. It’s also valuable to post critiques of these ideas.
We’re using a loose framework of pre-seed grants of $200k–$2m, seed grants of $2m–$20m, and bigger bets of $20m–$200m or even more. I expect that we’ll often give out pre-seed grants to promising people with good ideas, even when their plans are not particularly fleshed out and they expect to pivot around a lot. We also fund generous compute budgets and competitive salaries.
The easiest way to apply for funding is to fill out this quick expression-of-interest form. If your EOI sounds promising we’ll invite you to submit a full proposal.
Full list of example orgs:
Independent assessment
Frontier lab accountability: Independent tracking, comparison, and review of frontier labs’ safety-relevant practices.
Misalignment red team: Discover misalignment properties in frontier models before the world finds out the hard way.
Control red team: Stress-test AI companies’ security and control protocols before a misaligned AI does.
Bio safeguards red team: Systematic red teaming of gene synthesis and AIxBio safeguards.
Independent AI auditors: Build the independent audit capacity necessary for AI regulation to succeed.
Investigations for secret loyalties: Determine whether companies can prevent a small group of people from bending AI to their own interests.
Misleadingness evaluations: Build evaluations that incentivize labs to make models less misleading on fuzzy tasks.
Research institutions
Oversight of CoT monitorability: Measure how monitorable frontier model chains of thought are, track if that property degrades, and identify ways to preserve it.
AI character evaluator: Improve our understanding of AIs’ behavioral tendencies and overall values.
Unclaimed research labs: Build a research lab around one of the many promising safety agendas still waiting for a team.
Training run fault injections: Construct misaligned models in vitro to map which training procedures contribute to misalignment.
Threat modeling institute: Develop and curate the best models of how catastrophic risks from AI could occur.
Alignment moonshots: Pioneer alignment techniques robust enough to trust with superintelligent AI.
Evidence generation
Rogue incident detection: Independent detection of real-world AI-driven attacks.
AI incident investigations: Investigate reports of misalignment or misuse and determine their implications.
Frontier capabilities observatory: Provide decision-makers with clear demonstrations of what frontier AI models can do.
State of AI risk: Publish canonical living literature reviews on what we know about different risks from advanced AI.
Biological uplift demos: Create and use demos to communicate frontier AI-bio uplift to policymakers.
AI security & verification
Agreement verification technology: Build the technology and protocols to enable and verify agreements on safe AI development.
Security capacity buildout: Build the talent pipelines and infrastructure to bring AI projects in the U.S. to SL4 and SL5.
AI integrity standards: Build security protocols that provide assurance an AI’s integrity has not been compromised.
Public goods
Think tank for AI labs: On-demand independent expertise for frontier AI safety and governance teams.
AI for decision-makers: Build AI tools that improve the quality of what could be the world’s most important decisions.
AI uplift for AI safety: Help the independent ecosystem make good use of frontier capabilities.
Compute cluster for AI safety: Build and run a GPU cluster that serves the AI safety nonprofit ecosystem.
Datasets for AI safety: Build datasets and environments that differentially improve AI safety capabilities.
Grand awards for AI safety: Run competitions for important work in AI safety.
Talent acceleration
Senior talent headhunting: Recruit senior or specialized talent to work on important areas in AI safety.
Talent matchmaking: Build infrastructure to fill important roles in the AI safety ecosystem.
Mid-career entry points: Expand opportunities for experienced professionals to join AI safety orgs.
AI context accelerator: Help experienced professionals build the context and networks for AI safety leadership roles.
Philanthropic startup incubator: Build the infrastructure to launch and scale cohorts of new AI safety projects.
Fiscal sponsors: Build fiscal sponsorship capacity to absorb new organizations over the next 1-2 years.
Convenings
Cross-sector convenings: Facilitate engagement among key groups working on frontier AI.
Subfield forums: Run events aimed at focusing attention on specific areas within AI safety.
Events as a service: Make sure every organization that wants to run an event can do so easily.
Some of these initiatives might not require founding an org, and might be better described as projects, but I’ve used ‘org’ everywhere for simplicity.
You can also apply for funding for an existing org.
Up for a call to discuss human intelligence amplification? (I applied but I am worried that the application reviewer may make hasty judgements.)
(As a CG employee) To my knowledge, none of the teams at CG fund HIA or have plans to do so.
Just for common knowledge, while Dustin is no longer blocking funding to rationalist-adjacent organizations, CG has still rejected funding Lightcone Infrastructure (despite Dustin actively wanting to make a grant to us[1]). I find it frustrating to say that “funding isn’t really a bottleneck” while you continue to block funding to a lot of the historically most effective organizations (by the lights of your own staff!).
Funding continues to be a non-trivial bottleneck for many of the best organizations. I continue to be highly concerned about the pressures and selection effects that apply when an organization is trying to get funding from CG.
Which he indeed ended up doing directly from Good Ventures after (as far as I know) CG-leadership vetoed grant recommendations by its staff to us (and which I am very grateful to Dustin for doing).
Fair enough, seems true that funding is still a bottleneck for orgs that CG doesn’t fund, and maybe it’s clumsy/insensitive to throw that phrase around given that.
There are, at least, many impactful things one can do that are not currently bottlenecked on CG’s willingness to fund them, but on the availability of competent people to do them.
Hi Matt! What is Coefficient’s view on slowing down and stopping AI? Do you fund people working on this?
(Unsure how helpful pointing this out is given that CG has probably gotten this critique before, but pointing it out anyways on the off chance it is helpful.)
“The range of AI safety work that CG is willing to fund is pretty broad these days, too.”
On the contrary it seems like your list of examples has a pretty narrow range. Much of it seems to be underpinned by a worldview that is something like “LLMs scale smoothly and predictably to AGI within the frontier labs and within a few years to a decade” (with some aspects of this being more or less load-bearing than others). I get that this view looks quite favorable right now, and I get that it’s hard to find ideas for deploying a large amount of funding under alternatives to this view, but I’d love to see more effort put into seeking out and funding weird alternative research programs and projects that help the field be diversified enough to be robust to some aspects of this worldview being wrong. Even if this view is entirely correct, it still seems to me that these projects are too highly correlated with each other and more decorrelated bets can be made.
Funding Tsvi’s HIA project would be a good start.
If CG in fact agrees with this idea, then signaling this more clearly would be ideal.
Just to share my thoughts after seeing the announcement and how I’ve been thinking transitioning from my previous work (in case it’s helpful to others in a similar boat):
I intend to apply, though I expect it will take me ~2-3 months to prepare the application (not the actual writing of it). In my case, my guess is that I will need to dedicate time to find co-founders and advisors before my application will be taken seriously (which is how I see what “funding is not a bottleneck, but talent is” is pointing to). Multiple people have told me that what I’m pursuing fits exactly into what CG and other funders want, but my guess is that I need to form a strong team first (perhaps with those who have gained more status in the field) and some preliminary related work.
As a result, I’m hoping to get short-term funding so I can even have the runway to apply properly. I was hoping to apply for much more funding faster so that I could then have the funding and take my time work trialing co-founders, but I hopefully I can get short-term funding to fill the gap and I don’t remain unpaid for months. I am mostly anxious about spending months preparing for this while not being paid and draining all my funds in the last ~1.5 years pursuing an AI safety startup (because I was hoping to bring in more money to AI safety since there wasn’t much philanthropic spending), living very frugally.
Anyway, thank you for opening the war chest!
You might be overestimating how developed a project has to be to get pre-seed funding. I can think of several recent grants we’ve made to very early-stage projects. So I think it’s probably worth submitting an EOI now
> In my case, my guess is that I will need to dedicate time to find co-founders and advisors before my application will be taken seriously (which is how I see what “funding is not a bottleneck, but talent is” is pointing to).
Good co-founders and advisors can be help in getting funding, but by “talent is a bottleneck” I just meant “we could really do with more great people eager to start projects”.
Thanks for the response!
I’ll consider applying sooner. My guess has been that the bar for talent will be high, and asking for funding anywhere near what I’d like would become a bottleneck because I don’t yet have enough ‘talent’. I’ve been assuming I can ask for much more in ~3 months, after I have a strong team (vs. telling funders, “I’m actively looking for co-founders who will fill roles x and y”). Maybe the happy-medium solution is to ask for less money ASAP, then apply for more funding every few months as we seek to grow.
One recommendation I have is to submit both a high and a low funding ask.
Consider applying to a career transition grant.
I’ve been working in AI safety for years. My guess is this isn’t meant for someone like me. (I did apply for a Rapid Grant, however.)
Edit: Tejas DMed after checking with BlueDot, and they said: “yes, have made multiple of these grants. seems great if there is high potential upside.”
What kind of submission timeline is helpful for a pre-seed EOI? Is it something where it’s beneficial to have it in within the next few days or is it okay to take a few weeks to solidify the proposal more?
It’s ok to take a few weeks for sure