Thanks for sharing, this is extremely important context—I’m way more ok with dual use threats from a company actively trying to reduce bio risk from AI who seem to have vaguely reasonable threat models, than just reckless gain of function people with insane threat models. It’s much less clear to me how much risk is ok to accept from projects actively doing reasonable things to make it better, but it’s clearly non zero (I don’t know if this place is actually doing reasonable things, but Mikhail provides no evidence against)
I think it was pretty misleading for Mikhail not to include this context in the original post.
Uhm yeah valid I guess the issue was illusion of transparency: I mostly copied the original post from my tweet, which was quote-tweeting the announcement, and I didn’t particularly think about adding more context because had it cached that the tweet is fine (I checked with people closely familiar with RQB before tweeting, and it did include all of the context by virtue of quote-tweeting the original announceemnt) and when posting to lw did not realize i’m not directly adding all of the context that was included in the tweet if people don’t click on the link.
Added the context to the original post.
Separately, I think an issue is that they’re incredibly non-transparent about what they’re doing and have been somewhat misleading in their responses to my tweets and not answering any of the questions.
Like, I can see a case for doing gain-of-function research responsibly to develop protection against threats (vaccines, proteins that would bind for viruses, etc.), but this should include incredible transparency, strong security (BSL & computer security & strong guardrails around what exactly AI models have automated access to), etc.
Separately, I think an issue is that they’re incredibly non-transparent about what they’re doing and have been somewhat misleading in their responses to my tweets and not answering any of the questions.
I can’t really fault them for not answering or being fully honest, from their perspective you’re a random dude who’s attacking them publicly and trying to get them lots of bad PR. I think it’s often very reasonable to just not engage in situations like that. Though I would judge them for outright lying
That’s somewhat reasonable. (They did engage though: made a number of comments and quote-tweeted my tweet, without addressing at all the main questions.)
Sure, but there’s a big difference between engaging in PR damage control mode and actually seriously engaging. I don’t take them choosing to be in the former as significant evidence of wrong doing
Since 2016, I have been building HelixNano, a clinical stage biotech (and still my main gig), with Nikolai Eroshenko. Recently, HelixNano teamed up with OpenAI to push AI bio’s limits. To our surprise, we saw models invent genuinely new wet lab methods (publication soon).
We got super excited. There was a path to superhuman drug designers. But we couldn’t ignore the shadow of superhuman virus designers. A world with breakthrough AI drugs can’t exist without new biological defenses. We spun out Red Queen Bio to build them.
Based on this, they didn’t need to set up a new company. They already had an existing biotech company that was focused on its own research, when they realized that “oh fuck, based on our current research things could get really bad unless someone does something”… and then they went Heroic Responsibility and spun out a whole new company to do something, rather than just pretending that no dangers existed or making vague noises and asking for government intervention or something.
It feels like being hostile toward them is a bit Copenhagen Ethics, in that if they hadn’t tried to do the right thing, it’s possible that nobody would have heard about this and things would have been much easier for them. But since they were thinking about their consequences of their research and decided to do something about it and said that in public, they’re now getting piled on for not answering every question they’re asked on X. (And if I were them, I might also have concluded that the other side is so hostile that every answer might be interpreted in the worst possible light and that it’s better not to engage.)
Thanks for sharing, this is extremely important context—I’m way more ok with dual use threats from a company actively trying to reduce bio risk from AI who seem to have vaguely reasonable threat models, than just reckless gain of function people with insane threat models. It’s much less clear to me how much risk is ok to accept from projects actively doing reasonable things to make it better, but it’s clearly non zero (I don’t know if this place is actually doing reasonable things, but Mikhail provides no evidence against)
I think it was pretty misleading for Mikhail not to include this context in the original post.
Uhm yeah valid I guess the issue was illusion of transparency: I mostly copied the original post from my tweet, which was quote-tweeting the announcement, and I didn’t particularly think about adding more context because had it cached that the tweet is fine (I checked with people closely familiar with RQB before tweeting, and it did include all of the context by virtue of quote-tweeting the original announceemnt) and when posting to lw did not realize i’m not directly adding all of the context that was included in the tweet if people don’t click on the link.
Added the context to the original post.
Separately, I think an issue is that they’re incredibly non-transparent about what they’re doing and have been somewhat misleading in their responses to my tweets and not answering any of the questions.
Like, I can see a case for doing gain-of-function research responsibly to develop protection against threats (vaccines, proteins that would bind for viruses, etc.), but this should include incredible transparency, strong security (BSL & computer security & strong guardrails around what exactly AI models have automated access to), etc.
Thanks for adding the context!
I can’t really fault them for not answering or being fully honest, from their perspective you’re a random dude who’s attacking them publicly and trying to get them lots of bad PR. I think it’s often very reasonable to just not engage in situations like that. Though I would judge them for outright lying
That’s somewhat reasonable. (They did engage though: made a number of comments and quote-tweeted my tweet, without addressing at all the main questions.)
Sure, but there’s a big difference between engaging in PR damage control mode and actually seriously engaging. I don’t take them choosing to be in the former as significant evidence of wrong doing
Agree; I’d also like to emphasize this part:
Based on this, they didn’t need to set up a new company. They already had an existing biotech company that was focused on its own research, when they realized that “oh fuck, based on our current research things could get really bad unless someone does something”… and then they went Heroic Responsibility and spun out a whole new company to do something, rather than just pretending that no dangers existed or making vague noises and asking for government intervention or something.
It feels like being hostile toward them is a bit Copenhagen Ethics, in that if they hadn’t tried to do the right thing, it’s possible that nobody would have heard about this and things would have been much easier for them. But since they were thinking about their consequences of their research and decided to do something about it and said that in public, they’re now getting piled on for not answering every question they’re asked on X. (And if I were them, I might also have concluded that the other side is so hostile that every answer might be interpreted in the worst possible light and that it’s better not to engage.)