I agree that any way this leads to a pulled punch is a failure. Let’s look at the effect on Detection and AI separately.
On the Detection side (the division I lead), punching hasn’t ever been in our mission, and I don’t think that’s where we should fit in the ecosystem. We’re building an early warning system, with a focus on engineered pathogens that could spread widely before they’d otherwise be detected. We’ve done a very small amount of policy advocacy, in the sense of “here’s why it’s good for detection systems to be built in a pathogen-agnostic way” and “if you had $X to spend on a detection system here’s the sensitivity we think you could get”, but our work is fundamentally focused on the technology, as researchers and implementers.
On the AI side (the division Jasper leads), I do think quite a bit of their work can be characterized as a kind of punching. But in a non-central way: it’s not aggression but that their evaluatory role requires telling the truth whatever it may be. That is, they need to accurately evaluate models and share their findings, and when that means saying truths that are uncomfortable or damaging to an AI company, that is absolutely their role. Incentives like “model developers are less likely to give you special access to future models if you’re too negative in what you say about the current ones” push the wrong way, and I think regulation that requires this access would be fantastic. This isn’t a part of SecureBio I am close to, being focused on Detection, but it does look to me like the SBAI folks are good individually and institutionally at resisting these incentives. This is a fraught place to be, but I don’t see a better option.
This is why I think it’s key that none of this grant can go to support AI work. We have it in a separate account, that we charge Detection expenses to. This is how we wanted it, and OAIF was fully on board with it (and made it a grant requirement). It’s also key that there not be pressure from Detection to AI to go easy on OpenAI, and we’ve intentionally avoided avenues where that could flow. I don’t get to (or want to!) review AI evals before they go out, and neither does anyone else on Detection. “Could this hurt Detection’s ability to get further money from OAIF” is not the kind of thing Jasper’s org would consider, and I’ve explicitly confirmed to Jasper that I agree this is how his org should approach it.
As for making them look good, I think a grant to Detection does significantly less to make OpenAI look good than many of their other options. We’re still a bit weird: we’re defending against engineered pathogens, something that a lot of people dismiss as science fiction. They could be giving local kids free admission to museums, supporting rural EMS, rebuilding schools damaged by storms, etc, all for much more positive publicity.
Cross-posting a comment reply from substack:
I agree that any way this leads to a pulled punch is a failure. Let’s look at the effect on Detection and AI separately.
On the Detection side (the division I lead), punching hasn’t ever been in our mission, and I don’t think that’s where we should fit in the ecosystem. We’re building an early warning system, with a focus on engineered pathogens that could spread widely before they’d otherwise be detected. We’ve done a very small amount of policy advocacy, in the sense of “here’s why it’s good for detection systems to be built in a pathogen-agnostic way” and “if you had $X to spend on a detection system here’s the sensitivity we think you could get”, but our work is fundamentally focused on the technology, as researchers and implementers.
On the AI side (the division Jasper leads), I do think quite a bit of their work can be characterized as a kind of punching. But in a non-central way: it’s not aggression but that their evaluatory role requires telling the truth whatever it may be. That is, they need to accurately evaluate models and share their findings, and when that means saying truths that are uncomfortable or damaging to an AI company, that is absolutely their role. Incentives like “model developers are less likely to give you special access to future models if you’re too negative in what you say about the current ones” push the wrong way, and I think regulation that requires this access would be fantastic. This isn’t a part of SecureBio I am close to, being focused on Detection, but it does look to me like the SBAI folks are good individually and institutionally at resisting these incentives. This is a fraught place to be, but I don’t see a better option.
This is why I think it’s key that none of this grant can go to support AI work. We have it in a separate account, that we charge Detection expenses to. This is how we wanted it, and OAIF was fully on board with it (and made it a grant requirement). It’s also key that there not be pressure from Detection to AI to go easy on OpenAI, and we’ve intentionally avoided avenues where that could flow. I don’t get to (or want to!) review AI evals before they go out, and neither does anyone else on Detection. “Could this hurt Detection’s ability to get further money from OAIF” is not the kind of thing Jasper’s org would consider, and I’ve explicitly confirmed to Jasper that I agree this is how his org should approach it.
As for making them look good, I think a grant to Detection does significantly less to make OpenAI look good than many of their other options. We’re still a bit weird: we’re defending against engineered pathogens, something that a lot of people dismiss as science fiction. They could be giving local kids free admission to museums, supporting rural EMS, rebuilding schools damaged by storms, etc, all for much more positive publicity.