It seems like you’re not being clear about how you are thinking about the cases, or are misusing some of the terms. Nash Equilibria exist in zero-sum games, so those aren’t different things. If you’re familiar with how to do game theory, I think you should carefully set up what you claim the situation is in a payoff matrix, and then check whether, given the set of actions you posit people have in each case, the scenario is actually a Nash equilibrium in the cases you’re calling Nash equilibrium.
Davidmanheim
Karma: 4,448
...but there are a number of EAs working on cybersecurity in the context of AI risks, so one premise of the argument here is off.
And a rapid response site for the public to report cybersecurity issues and account hacking generally would do nothing to address the problems that face the groups that most need to secure their systems, and wouldn’t even solve the narrower problem of reducing those hacks, so this seems like the wrong approach even given the assumptions you suggest.
I agree that your question is weird and confused, and agree that if that were the context, my post would be hard to understand. But I think it’s a bad analogy! That’s because there are people who have made analogies between AI and Bio very poorly, and it’s misleading and leading to sloppy thinking. In my experience seeing discussions on the topic, either the comparisons are drawn carefully and the relevant dissimilarities are discussed clearly, or they are bad analogies.
To stretch your analogy, if the context were that I’d recently heard people say “Steve and David are both people I know, and if you don’t like Steve, you probably won’t like David,” and also “Steve and David are both concerned about AI risks, so they agree on how to discuss the issue,” I’d wonder if there was some confusion, and I’d feel comfortable saying that in general, Steve is an unhelpful analog for David, and all these people should stop and be much more careful in how they think about comparisons between us.
I agree with you that analogies are needed, but they are also inevitably limited. So I’m fine with saying “AI is concerning because its progress is exponential, and we have seen from COVID-19 that we need to intervene early,” or “AI is concerning because it can proliferate as a technology like nuclear weapons,” or “AI is like biological weapons in that countries will pursue and use these because they seem powerful, without appreciating the dangers they create if they escape control.” But what I am concerned that you are suggesting is that we should make the general claim “AI poses uncontrollable risks like pathogens do,” or “AI needs to be regulated the way biological pathogens are,” and that’s something I strongly oppose. By ignoring all of the specifics, the analogy fails.
In other words, “while I think the disanalogies are compelling, comparison can still be useful as an analytic tool—while keeping in mind that the ability to directly learn lessons from biorisk to apply to AI is limited by the vast array of other disanalogies.”
I said:
disanalogies listed here aren’t in and of themselves reasons that similar strategies cannot sometimes be useful, once the limitations are understood. For that reason, disanalogies should be a reminder and a caution against analogizing, not a reason on its own to reject parallel approaches in the different domains.
You seem to be simultaneously claiming that I had plenty of room to make a more nuanced argument, and then saying you think I’m saying something which exactly the nuance I included seems to address. Yes, people could cite the title of the blog post to make a misleading claim, assuming others won’t read it—and if that’s your concern, perhaps it would be enough to change the title to “Biorisk is Often an Unhelpful Analogy for AI Risk,” or “Biorisk is Misleading as a General Analogy for AI Risk”?
I agree that we do not have an exact model for anything in immunology, unlike physics, and there is a huge amount of uncertainty. But that’s different than saying it’s not well-understood; we have clear gold-standard methods for determining answers, even if they are very expensive. This stands in stark contrast to AI, where we don’t have the ability verify that something works or is safe at all without deploying it, and even that isn’t much of a check on its later potential for misuse.
But aside from that, I think your position is agreeing with mine much more than you imply. My understanding is that we have newer predictive models which can give uncertain but fairly accurate answers to many narrow questions. (Older, non-ML methods also exist, but I’m less familiar with them.) In your hypothetical case, I expect that the right experts can absolutely give indicative answers about whether a novel vaccine peptide is likely or unlikely to have cross-reactivity with various immune targets, and the biggest problem is that it’s socially unacceptable to assert confidence in anything short of tested and verified case. But the models can get, in the case of the Zhang et al paper above, 70% accurate answers, which can help narrow the problem for drug or vaccine discovery, then they do need to be followed with in vitro tests and trials.
I’m arguing exactly the opposite; experts want to make comparisons carefully, and those trying to transmit the case to the general public should, at this point, stop using these rhetorical shortcuts that imply wrong and misleading things.
On net, the analogies being used to try to explain are bad and misleading.
I agree that I could have tried to convey a different message, but I don’t think it’s the right one. Anyone who wants to dig in can decide for themselves, but you’re arguing that ideal reasoners won’t conflate different things and can disentangle the similarities and differences, and I agree, but I’m noting that people aren’t doing that, and others seem to agree.
I don’t understand why you disagree. Sure, pathogens can have many hosts, but hosts generally follow the same logic as for humans in terms of their attack surface being static and well adapted, and are similarly increasingly understood.
Biorisk is an Unhelpful Analogy for AI Risk
That doesn’t seem like “consistently and catastrophically,” it seems like “far too often, but with thankfully fairly limited local consequences.”
BSL isn’t the thing that defines “appropriate units of risk”, that’s pathogen risk-group levels, and I agree that those are are problem because they focus on pathogen lists rather than actual risks. I actually think BSL are good at what they do, and the problem is regulation and oversight, which is patchy, as well as transparency, of which there is far too little. But those are issues with oversight, not with the types of biosecurity measure that are available.
If you’re appealing to OpenPhil, it might be useful to ask one of the people who was working with them on this as well.
And you’ve now equivocated between “they’ve induced an EA cause area” and a list of the range of risks covered by biosecurity—not what their primary concerns are—and citing this as “one of them.” I certainly agree that biosecurity levels are one of the things biosecurity is about, and that “the possibility of accidental deployment of biological agents” is a key issue, but that’s incredibly far removed from the original claim that the failure of BSL levels induced the cause area!
I mean, I’m sure something more restrictive is possible.
But what? Should we insist that the entire time someone’s inside a BSL-4 lab, we have a second person who is an expert in biosafety visually monitoring them to ensure they don’t make mistakes? Or should their air supply not use filters and completely safe PAPRs, and feed them outside air though a tube that restricts their ability to move around instead? (Edit to Add: These are already both requires in BSL-4 labs. When I said I don’t know of anything more restrictive they could do, I was being essentially literal—they do everything including quite a number of unreasonable things to prevent human infection, short of just not doing the research.)
Or do you have some new idea that isn’t just a ban with more words?
“lists of restrictions” are a poor way of managing risk when the attack surface is enormous
Sure, list-based approaches are insufficient, but they have relatively little to do with biosafety levels of labs, they have to do with risk groups, which are distinct, but often conflated. (So Ebola or Smallpox isn’t a “BSL-4” pathogen, because there is no such thing. )
I just meant “gain of function” in the standard, common-use sense—e.g., that used in the 2014 ban on federal funding for such research.
That ban didn’t go far enough, since it only applied to 3 pathogen types, and wouldn’t have banned what Wuhan was doing with novel viruses, since that wasn’t working with SARS or MERS, it was working with other species of virus. So sure, we could enforce a broader version of that ban, but getting a good definition that’s both extensive enough to prevent dangerous work and that doesn’t ban obviously useful research is very hard.
Having written extensively about it, I promise you I’m aware. But please, tell me more about how this supports the original claim which I have been disagreeing with, that these class of incidents were or are the primary concern of the EA biosecurity community, the one that led to it being a cause area.
The OP claimed a failure of BSL levels was the single thing that induced biorisk as a cause area, and I said that was a confused claim. Feel free to find someone who disagrees with me here, but the proximate causes of EAs worrying about biorisk has nothing to do with BSL lab designations. It’s not BSL levels that failed in allowing things like the soviet bioweapons program, or led to the underfunded and largely unenforceable BWC, or the way that newer technologies are reducing the barriers to terrorists and other being able to pursue bioweapons.
I did not say that they didn’t want to ban things, I explicitly said “whether to allow certain classes of research at all,” and when I said “happy to rely on those levels, I meant that the idea that we should have “BSL-5” is the kind of silly thing that novice EAs propose that doesn’t make sense because there literally isn’t something significantly more restrictive other than just banning it.
I also think that “nearly all EA’s focused on biorisk think gain of function research should be banned” is obviously underspecified, and wrong because of the details. Yes, we all think that there is a class of work that should be banned, but tons of work that would be called gain of function isn’t in that class.
BSL levels, which have failed so consistently and catastrophically they’ve induced an EA cause area,
This is confused and wrong, in my view. The EA cause area around biorisk is mostly happy to rely on those levels, and unlike for AI, the (very useful) levels predate EA interest and give us something to build on. The questions are largely instead about whether to allow certain classes of research at all, the risks of those who intentionally do things that are forbiddn, and how new technology changes the risk.
and then the 2nd AI pays some trivial amount to the 1st for the inconvenience
Completely as an aside, coordination problems among ASI don’t go away, so this is a highly non trivial claim.
Most of this seems to be subsumed in the general question of how do you do research, and there’s lot of advice, but it’s (ironically) not at all a science. From my limited understanding of what goes on in the research groups inside these companies, it’s a combination of research intuition, small scale testing, checking with others and discussing the new approach, validating your ideas, and getting buy-in from people higher up that it’s worth your and their time to try the new idea. Which is the same as research generally.
At that point, I’ll speculate and assume whatever idea they have is validated in smaller but still relatively large settings. For things like sample efficiency, they might, say, train a GPT-3 size model, which now cost only a fraction of the researcher’s salary to do. (Yes, I’m sure they all have very large compute budgets for their research.) If the results are still impressive, I’m sure there is lots more discussion and testing before actually using the method in training the next round of frontier models that cost huge amounts of money—and those decisions are ultimately made by the teams building those models, and management.