No required rubric for evaluating applications
FWIW, I don’t know what rubric this is referencing for SFF S-Process rounds.
Broadly speaking, to evaluate a proposal a Recommender needs to make a Marginal Value Function and write some kind of note. We also ask Recommenders to provide some amount of positive or negative feedback to orgs they evaluated.
Furthermore, I want to remove any requirement by evaluators to look comprehensively at all applications submitted to a grant round.
In SFF S-Process rounds, Recommenders are required to skim or familiarize themselves with all applications that are included in the round, but they’re certainly not asked to evaluate or look comprehensively at them.
Second, I intend to remove requirements for every grant to first receive a speculation grant in order to be considered in a grant round.
In SFF S-Process rounds, Recommenders are allowed to whitelist applications that didn’t receive a Speculation Grant, and those applications are considered in the round. The only limits on that (iirc) are that the application has to have been submitted before that round’s application deadline, and a Recommender has to evaluate any application that they whitelist.
SFF has no institutional knowledge
SFF is reinstantiated with a new composition once per year
These are both overstatements.
The Recommender pool is reinstantiated every year with a new composition, but there are typically 3-6 returning Recommenders (out of 6-12 total). Institutional knowledge also carries over year-to-year through paid staff at SFC which help run SFF programs, and the persistent group of 30-40 Speculators who evaluate grant proposals prior to S-Process rounds.
Thanks for writing this.
On the data loss issue, since that’s something we take seriously:
Usually my changes were autosaved but regularly they were just deleted.
We’re looking into the autosave issue now that it’s been brought to our attention. Fortunately we architected the app to record every edit someone makes — in the edit log we found one instance where one of your notes was reverted back to match a previous version, and three instances where one of your notes was truncated. I’ll get those diffs shared with you on Slack and you can restore the earlier versions of those notes if you want.
Factual corrections:
Each application was given an AI-generated name; many were convoluted, confusing, ambiguous, and/or incorrect.
Proposal short names were manually assigned – it’s hard to make distinctive unique identifiers when so many organizations have similar names.
Lack of division of labor.
Recommenders did divide up which proposals they’d evaluate, using the tooling in the app for marking interest and claiming proposals for deeper investigation.
Every recommender evaluates every application (skipping ones they know they’re not interested in)
Recommenders were asked to skim and familiarize themselves with all ~200 proposals, but were only required to form opinions on 50.
And these days to be considered you have to submit another application explaining why you need money quickly (even if you don’t) and receive a “speculation grant” based on that application.
The application for Speculation Grants is part of the same Google Form as the main funding application, and the question about why you need money quickly is marked as optional. (“If you don’t have any particular reason(s) that your organization needs expedited funds, you can leave this question blank.”)
it apparently doesn’t do user interviews
We do conduct user interviews, though of course all else equal it would be valuable for us to conduct more.
Additional context:
the stuff we were required to do took much more than 30 hours
There were ~200 proposals compared to the ~120 in the 2025 round, so Recommenders were asked to skim an additional ~80 proposals compared to last year. The required workload was otherwise the same as last year.
This was not feasible (and it was initially impossible to complete due to software bugs).
The bug I’m aware of that made it impossible to complete the task of skimming all ~200 proposals affected 3 of those proposals and was fixed within a day of it being reported.
The process is slow: it takes five months from application deadline to funding decisions (and it only runs once per year) (but some small projects can be funded quickly by “speculation granters”).
Speculation Grants exist to speed up distribution of funding; Speculators approved over $8M in Speculation Grants in April and May of this year.
The application is exceptionally long and annoying
Reducing the length and complexity of the application would lead to more low-effort submissions on the margin, which would compete for attention with high-effort submissions. We’re working on approaches for making improvements to the applicant experience while reducing the flood of applications for Speculators and Recommenders to deal with.
Building excellent, fully-featured, bug-free software is hard, and SFC doesn’t consistently meet that bar. That’s why we have Slack channels for reporting bugs and making feature requests, and why we set aside time at the start of Recommender meetings to ask if there have been technical difficulties. Over the next few months as the grant round wraps up, we’ll collect feedback from Recommenders, conduct user interviews, and figure out the highest impact improvements we can make to our tooling and our processes.
Ah I’ve got a guess now: there’s a rubric we added to give AI suggestions of how to improve their notes, with a checklist of 7 features associated with notes that have been most useful in the past.
That’s only one of like 5-10 ways we incorporated AI this year, and the rubric is a nudge rather than a requirement, but I get how someone could have experienced it as “this rubric is the required way to evaluate proposals, yuck.”