SFF is very suboptimal
Update: Ethan, who runs (SFC, the org that runs) SFF, replied here.
I recently served as a recommender in SFF’s annual funding round (grants will be decided and announced in September). I’m deeply grateful for SFF’s funders, and I hope more AI safety donors appear in the future.
Unfortunately, the SFF experience is bad for both applicants and recommenders, it’s slow, and it lacks some other desiderata. (The actual funding decisions are moderately good by my lights — probably over half of the value of grants that a better system with the same budget would produce, but substantial room for improvement.) The basic idea of the s-process—several recommenders compete to convince the funders to fund their recommendations each round, and so several different perspectives each get the best things by their lights funded and there’s little delegation of power—is attractive. But SFF’s implementation is bad.
My experience as a recommender
Recommenders were told we would only have to spend 30 hours, but the stuff we were required to do took much more than 30 hours, and doing a good job took even longer.[1]
[Cherry-picked / unusually legible] example suggesting that the time commitment was more than SFF said: we were asked to spend one hour glancing at the top of some 150-200 applications and, for each application, score our interest in investigating from 0 to 4. This was not feasible (and it was initially impossible to complete due to software bugs).
The custom software (to let recommenders process applications and make recommendations) was bad — it had lots of bugs and it just failed to make evaluating applications easy.
The information that the software thinks I want isn’t the information that I actually want.
Navigating within and (especially) between applications was hard.
The application-review workflow was just annoying/hard/clunky in many ways.
Each application was given an AI-generated name; many were convoluted, confusing, ambiguous, and/or incorrect. (Overall substantially worse than the names provided by the applicants; often the applicant-provided names were hidden so we could only see the bad AI names.)
Because applications are so long, I think I read zero applications in full. Recommenders often engaged with AI summaries of applications. This is inefficient; it would be better in many ways if applications were shorter and recommenders read the applications.
Usually my changes were autosaved but regularly they were just deleted.
Several more I’m forgetting.[2]
Many recommenders were ill-equipped for the task in various ways — lacking experience, lacking info on grantmaking, misunderstanding the utility functions they create.
Rather than recommending a certain grant size to each applicant, recommenders create a utility function for each applicant representing the value of giving them various amounts of money, and the utility functions automatically determine grant sizes. I believe there’s multiple cases where, at least before I intervened, a recommender’s utility functions clearly didn’t represent their views. For example, one (smart, experienced) recommender’s utility functions initially said that for each applicant, marginal funding hits zero value by $1.7M to the applicant; after I expressed disagreement, they changed their curves to show that some applicants would produce positive value with marginal funding beyond $1.7M. Unfortunately most issues are more subtle.
Some issues could be improved by choosing different recommenders. Some issues could be improved by offering better resources to recommenders.
Several recommenders were predictably bad choices by my lights. (In theory it could fine if an s-process has some bad recommenders—in theory the funders can just disregard them—but I expect recommender selection is important for SFF.)
Lack of division of labor. Every recommender evaluates every application (skipping ones they know they’re not interested in), and you can see other recommenders’ notes and ask them questions but overall division-of-labor mechanisms were poor.
Lack of institutional knowledge, lack of info on particular orgs, and other structural-ish problems.
For structural reasons there’s nobody responsible for paying attention to an applicant year-to-year. A more solvable problem is that by default recommenders don’t have access to info that professional grantmakers have about various orgs. Also most recommenders have never done this before (like me!) and there’s little training/resources.
Applicants can ask for a matching pledge in addition to a grant. This is a fine idea but the implementation is a mess.[3] And I’d guess most of the recommenders don’t really understand how it works (and details were only explained to us at the very end of the process) and this will lead to funding recommendations that don’t match the recommenders’ views.
My impression is that other recommenders also had a bad experience. If SFF surveys the six main track recommenders in the next few weeks, for the five besides me, I predict at least three will say they had an overall negative experience (but most others are less pessimistic than me).
Applicant experience
The application is exceptionally long and annoying; I think SFF is an outlier in this dimension; it causes those who do apply to waste time and other promising projects to not apply at all. The process is slow: it takes five months from application deadline to funding decisions (and it only runs once per year) (but some small projects can be funded quickly by “speculation granters”). And these days to be considered you have to submit another application explaining why you need money quickly (even if you don’t) and receive a “speculation grant” based on that application.
Some desiderata
Making good grants: meh.[4]
Part of this is real disagreements between me and other recommenders and/or funders, but a big part is structural stuff including “SFF recommenders are insufficiently sensitive to applicants’ runways or funging with GV[5] and inflexible funders” (since the software doesn’t make this salient and recommenders aren’t trained to care about it), issues with recommenders not knowing how to express their views in the software (especially for matching pledges), recommenders lacking some kinds of info, maybe “SFF isn’t set up to deal with adverse selection,” and many of the aforementioned issues with the process (which just make recommenders’ recommendations less good).
Speed: poor.
Five months from application deadline to funding decisions. Some orgs don’t need funding quickly, but in many cases a project needs funding—e.g. runway for at least a year—in order to start or to scale up. Also facts and plans change in five months.
Flexibility: poor.
You can’t do private stuff, you can’t really coordinate with other funders, you can’t really steer grantees. See Considerations against s-process philanthropy for details on some desiderata — some of those issues are surmountable, but SFF doesn’t currently surmount them.
Predictability: meh to poor. (Less important, especially since SFF is a small fraction of total funding.)
Almost all applicants are pretty unable to predict how much funding they’ll receive. For structural reasons (including that the total budget and distribution between recommenders is only determined at the very end of the process), even recommenders have little sense of how much funding each applicant will receive. To some extent this is inevitable in a system designed to avoid delegation of power from the funder to the recommenders.
SFF has recently tried implementing some confusing policies on applicant eligibility, including policies aimed at avoiding funding a large fraction of “advocacy” projects. This has costs including wasting evaluation time and causing uncertainty/chaos for various groups of people in addition to the prospect of missing good grant opportunities.
(SFF, as an s-process, does seem to succeed at avoiding large-scale delegation of power (which results in various problems) and keeping the philanthropy responsive to the funders’ views.)
I’m not asserting that SFF is the worst AI safety funding institution. And relative to a vacuum—that is, if not for the effect of crowding out potential similar efforts—I believe SFF is substantially net-positive. I just wrote this post because I recently developed conviction that SFF is leaving lots of cheap value on the table.
This post is partially written as feedback for SFF, but I have little faith in SFF to improve. The status quo seems to be that SFF isn’t trying to be good (e.g. it had recommenders use software and processes that were clearly untested, and it apparently doesn’t do user interviews). So I think specific suggestions won’t go far without a general wakeup call.
habryka will soon launch a new s-process system. I am optimistic that it will be much better than SFF. If it is, I hope SFF’s funders switch to it.
- ^
I expected from the beginning that most recommenders would spend much more than 30 hours. I incorrectly thought I personally would be able to spend just 30 hours. I still think I could have made good recommendations in 30 hours, but (1) we were required to do specific things which I wouldn’t have done if I was trying to do my best in 30 hours (e.g. enter an evaluation for at least 50 applications) and (2) your budget is determined at the end of the process and is indirectly correlated with how much time you spend so if you want to have more impact you have to spend more time.
- ^
After writing this post I glanced at the relevant slack channel and yep, there’s much more. And glancing at that channel reminded me of some bugs that weren’t reported in that channel, including:
Text-editor tools (bullets, bold, italics) didn’t work in my browser
Applications with special characters in the name initially couldn’t be opened.
- ^
Claim not justified here but my impression is almost everyone agrees. I think several—perhaps most/all—recommenders’ views aren’t captured by their utility functions for matching pledges. For one, I expect several recommenders left the parameters at their default values, which may be systematically wrong. And applicants can’t really predict the implications of opting in.
- ^
To be clear, the funders haven’t yet decided what to fund this round. This is based on past rounds, my guesses about this round, and just a priori reason from how the process works.
- ^
“Funge with” (or “funge against”) means substitute with or displace or be displaced by. “GV” means Good Ventures. In many cases, GV is happy for flexible AI safety funders to not donate to the orgs it fully funds (and instead donate to the good projects that it doesn’t fund for various reasons).
Criticism is easy; doing things is hard and scary. I really appreciate everyone who does things to make the future much better, and (while I dislike the SFF product) that absolutely includes everyone who works for SFF. Praise should be much more salient than criticism, it’s messed up that it’s not, and I feel bad about contributing to that phenomenon via this post.
I’m glad you’re writing this comment. Speech acts are a thing, even if you don’t want them to be. Whether it is your intention or not, this post inflicts not-insubstantial reputational damage to SFF. Perhaps a private note would have been better?
One obvious good thing to say about SFF is that this post exists. I can’t recall ever seeing a grantmaker at a foundation write a public post about why the foundation is doing a poor job.
Yeah, SFF certainly avoids some failure modes that foundations are susceptible to, including because it’s just not set up like “the funder delegates power to the foundation.” Quoting habryka:
Thanks for posting this. I was in the same SFF round as Zach (I might be the person he’s talking about with the $1.7M allocations, although I think this made more sense in context) and endorse all of this.
I hope to write write up my thoughts on the process eventually, but SFF has suggested I hold off on writing about the thing I was most concerned about until they formulate a final policy in September, so I’ll avoid doing so until then except to +1 Zach.
Funny you should say this. As someone used to NIH grant applications (which can be over 100 pages long, full of mostly-boilerplate), I thought it was a lot easier in comparison! But SFF could definitely be improved. As it is, there’s a lot of near-duplicate questions.
Thanks for writing this.
On the data loss issue, since that’s something we take seriously:
We’re looking into the autosave issue now that it’s been brought to our attention. Fortunately we architected the app to record every edit someone makes — in the edit log we found one instance where one of your notes was reverted back to match a previous version, and three instances where one of your notes was truncated. I’ll get those diffs shared with you on Slack and you can restore the earlier versions of those notes if you want.
Factual corrections:
Proposal short names were manually assigned – it’s hard to make distinctive unique identifiers when so many organizations have similar names.
Recommenders did divide up which proposals they’d evaluate, using the tooling in the app for marking interest and claiming proposals for deeper investigation.
Recommenders were asked to skim and familiarize themselves with all ~200 proposals, but were only required to form opinions on 50.
The application for Speculation Grants is part of the same Google Form as the main funding application, and the question about why you need money quickly is marked as optional. (“If you don’t have any particular reason(s) that your organization needs expedited funds, you can leave this question blank.”)
We do conduct user interviews, though of course all else equal it would be valuable for us to conduct more.
Additional context:
There were ~200 proposals compared to the ~120 in the 2025 round, so Recommenders were asked to skim an additional ~80 proposals compared to last year. The required workload was otherwise the same as last year.
The bug I’m aware of that made it impossible to complete the task of skimming all ~200 proposals affected 3 of those proposals and was fixed within a day of it being reported.
Speculation Grants exist to speed up distribution of funding; Speculators approved over $8M in Speculation Grants in April and May of this year.
Reducing the length and complexity of the application would lead to more low-effort submissions on the margin, which would compete for attention with high-effort submissions. We’re working on approaches for making improvements to the applicant experience while reducing the flood of applications for Speculators and Recommenders to deal with.
Building excellent, fully-featured, bug-free software is hard, and SFC doesn’t consistently meet that bar. That’s why we have Slack channels for reporting bugs and making feature requests, and why we set aside time at the start of Recommender meetings to ask if there have been technical difficulties. Over the next few months as the grant round wraps up, we’ll collect feedback from Recommenders, conduct user interviews, and figure out the highest impact improvements we can make to our tooling and our processes.
Thank you for writing this! I’ve long been a fan of the SFF compared for the outputs it produces, but agree with the aspects of poor UX and slowness from the grantee side; it’s nice to hear about the recommended side.
Curious:
Does SFF pay its recommenders? Can you say how much?
Does SFF negotiate an impact split between its recommenders and its funders? (I’m not aware of one between itself and it’s grantees fwiw).
I wanted to suggest writing a new system until I read the last line