I suppose I was speaking too loosely—thank you for flagging that!
I don’t mean that it’s literally impossible to assess whether AI governance grants have been successful—only that doing so requires somewhat more deliberate effort than it does for most other types of grants, and that there is relatively less in the way of established infrastructure to support such measurements in the field of AI governance.
If you run an anti-malaria program, there’s a consensus about at least the broad strokes of what you’re supposed to measure (i.e., malaria cases), and you’ll get at least some useful information about that metric just from running your program and honestly recording what your program officers observe as they deliver medication. If your bed nets are radically reducing the incidence of malaria in your target population, then the people distributing those bed nets will probably notice. There is also an established literature on “experimental methods” for these kinds of interventions that tells us that we need to be taking measurements and how to do so and how to interpret them.
By contrast, if you’re slightly reducing the odds of an AI catastrophe, it’s not immediately obvious or agreed-upon what observable changes this ought to produce in the real world, and a grant funder isn’t very likely to notice those changes unless they specifically go and look for them. They’re also less likely to specifically go and look for them in an effective way, because the literature on experimental methods for politics is much less well-developed than the literature on experimental methods for public health.
My work so far has mostly been about doing the advocacy, rather than establishing better metrics to evaluate the impact of that advocacy. That said, in posts 1 and 7 of this sequence, I do suggest some starting points. I encourage funders to look at figures like the number of meetings had with politicians, the number of events that draw in a significant number of politicians, the number of (positive) mentions in mainstream ‘earned media’, the number of endorsements that are included in Congressional offices’ press releases, and the number (and relative importance) of edits made to Congressional bills.
If your work is focused on the executive or judicial branch instead of on Congress, you could adapt some of those metrics accordingly, e.g., edits to pending regulation or executive orders, or citations to your amicus curiae briefs in judicial opinions, and so on.
I suppose I was speaking too loosely—thank you for flagging that!
I don’t mean that it’s literally impossible to assess whether AI governance grants have been successful—only that doing so requires somewhat more deliberate effort than it does for most other types of grants, and that there is relatively less in the way of established infrastructure to support such measurements in the field of AI governance.
If you run an anti-malaria program, there’s a consensus about at least the broad strokes of what you’re supposed to measure (i.e., malaria cases), and you’ll get at least some useful information about that metric just from running your program and honestly recording what your program officers observe as they deliver medication. If your bed nets are radically reducing the incidence of malaria in your target population, then the people distributing those bed nets will probably notice. There is also an established literature on “experimental methods” for these kinds of interventions that tells us that we need to be taking measurements and how to do so and how to interpret them.
By contrast, if you’re slightly reducing the odds of an AI catastrophe, it’s not immediately obvious or agreed-upon what observable changes this ought to produce in the real world, and a grant funder isn’t very likely to notice those changes unless they specifically go and look for them. They’re also less likely to specifically go and look for them in an effective way, because the literature on experimental methods for politics is much less well-developed than the literature on experimental methods for public health.
My work so far has mostly been about doing the advocacy, rather than establishing better metrics to evaluate the impact of that advocacy. That said, in posts 1 and 7 of this sequence, I do suggest some starting points. I encourage funders to look at figures like the number of meetings had with politicians, the number of events that draw in a significant number of politicians, the number of (positive) mentions in mainstream ‘earned media’, the number of endorsements that are included in Congressional offices’ press releases, and the number (and relative importance) of edits made to Congressional bills.
If your work is focused on the executive or judicial branch instead of on Congress, you could adapt some of those metrics accordingly, e.g., edits to pending regulation or executive orders, or citations to your amicus curiae briefs in judicial opinions, and so on.
This is convincing!