Risk of misuse
The thought had never previously struck us.
This seems like one of the most tractable things to address to reduce AI risk.
If 5 years from now anyone developing AI or biotechnology is still not thinking (early and seriously) about ways their work could cause harm that other people have been talking about for years, I think we should consider ourselves to have failed.
The David Silver on it being okay to have AGIs with different goals part worried me because it sounded like he wasn’t at all thinking about the risk from misaligned AI. It seemed like he was saying we should create general intelligence regardless of its goals and values, just because it’s intelligent.
In this post Zvi doesn’t try to forecast how many infections/cases/deaths there would be in the US without this new strain (unless I missed it… it is a long post). Yet he really should, because doing so will lead one to realize that the US is likely going to be at or close to herd immunity by ~May-June anyway, so a new transmissible strain that becomes dominant in the US around that same period can’t plausibly make as huge of a difference as Zvi seems to be saying in this post.
Good Judgment’s median estimate for “How many total cases of COVID-19 in the U.S. will be estimated as of 31 March 2021?” is ~130M currently. And Good Judgment’s median estimate for “When will enough doses of FDA-approved COVID-19 vaccine(s) to inoculate 100 million people be distributed in the United States?” is ~May 1st currently. https://goodjudgment.io/covid/dashboard/
Assuming that 20% of vaccines go to people who had already been infected, this would mean that by May, approximately ~220M people (220M = ~140M + 0.8*100M) the US will be immune, or about 66% of the population. This could easily be higher or lower, but the point is that we’re going to be at or close to herd immunity by the time Zvi says this new viral strain would start becoming dominant in the US.
In short, the news would be much worse if this new viral strain had spread to the degree that it has now several months ago. But in reality, I think we’ll be at or close to herd immunity already by the time it becomes prominent, so it won’t make that much of a difference.
EDIT: I misread Zvi’s piece initially and mistakenly thought he wrote that that the new strain wouldn’t become dominant in the US until May. I now see that he says “Instead of that being the final peak and things only improving after that, we now face a potential fourth wave, likely cresting between March and May, that could be sufficiently powerful to substantially overshoot herd immunity.” Taking this view as true instead makes me see the new strain as significantly worse news: specifically, this two-month shift might be sufficient to make an additional ~10-15% of the population get infected/sick before herd immunity is reached. (I still think the post title is overblown, but still this is a significant update for me.)
On the subject of DeemMind and pausing AI development, I’d like to highlight Demis Hassabis’s remark on this topic in a DeepMind podcast interview a year ago:
‘Avengers assembled’ for AI Safety: Pause AI development to prove things mathematically
Hannah Fry (17:07):
You said you’ve got this sort of 20-year prediction and then simultaneously where society is in terms of understanding and grappling with these ideas. Do you think that DeepMind has a responsibility to hit pause at any point?
Demis Hassabis (17:24):
Potentially. I always imagine that as we got closer to the sort of gray zone that you were talking about earlier, the best thing to do might be to pause the pushing of the performance of these systems so that you can analyze down to minute detail exactly and maybe even prove things mathematically about the system so that you know the limits and otherwise of the systems that you’re building. At that point I think all the world’s greatest minds should probably be thinking about this problem. So that was what I would be advocating to you know the Terence Tao’s of this world, the best mathematicians. Actually I’ve even talked to him about this—I know you’re working on the Riemann hypothesis or something which is the best thing in mathematics but actually this is more pressing. I have this sort of idea of like almost uh ‘Avengers assembled’ of the scientific world because that’s a bit of like my dream.
To add some more emphasis to my point, because I think it deserves more emphasis:
Quoting the interview Jacy linked to:
Your paper also says that “[w]ithout being overly alarmist, this should serve as a wake-up call for our colleagues” — what is it that you want your colleagues to wake up to? And what do you think that being overly alarmist would look like?
We just want more researchers to acknowledge and be aware of potential misuse. When you start working in the chemistry space, you do get informed about misuse of chemistry, and you’re sort of responsible for making sure you avoid that as much as possible. In machine learning, there’s nothing of the sort. There’s no guidance on misuse of the technology.
I know I’m not saying anything new here, and I’m merely a layperson without ability to verify the truth of the claim I highlighted in bold above, but I do want to emphasize further:
I seems clear that changing the machine learning space so that it is like the chemistry space in the sense that you do get informed about ways machine learning can be misused and cause harm, is something that we should all push to make happen as soon as we can. (Also expanding the discussion of potential harm beyond harm caused from misuse to any harm related to the technology.)
Years ago I recall hearing Stuart Russell mention the analogy of how civil engineers don’t have a separate field for bridge safety; rather bridge safety is something all bridge designers are educated on and concerned about, and he similarly doesn’t want the field of AI safety to be separate from AI but wants all people working on AI to be educated on and concerned with risks from AI.
This is the same point I’m saying here, and I’m saying it again because it seems like the present machine learning space is still far from this point and we as a community really do need to devote more efforts to ensuring that we change this in the near future.
In a March 15th episode of the DeepMind podcast, Demis Hassabis said he has talked to Terence Tao about working on AI safety:
I always imagine that as we got closer to the sort of gray zone that you were talking about earlier, the best thing to do might be to pause the pushing of the performance of these systems so that you can analyze down to minute detail exactly and maybe even prove things mathematically about the system so that you know the limits and otherwise of the systems that you’re building. At that point I think all the world’s greatest minds should probably be thinking about this problem. So that was what I would be advocating to you know the Terence Tao’s of this world, the best mathematicians. Actually I’ve even talked to him about this—I know you’re working on the Riemann hypothesis or something which is the best thing in mathematics but actually this is more pressing. I have this sort of idea of like almost uh ‘Avengers assembled’ of the scientific world because that’s a bit of like my dream.
If we interpret his claim as an additional 0.5% wealth tax do you still think he is overstating his claim?
It’d be nice if Hassabis made another public statement about his views on pausing AI development and thoughts on the FLI petition. If now’s not the right time in his view, when is? And what can he do to help with coordination of the industry?
You’re probably already aware of this, but just in case not:
Demis Hassabis said the following about getting Terrence Tao to work on AI safety:
I always imagine that as we got closer to the sort of gray zone that you were talking about earlier, the best thing to do might be to pause the pushing of the performance of these systems so that you can analyze down to minute detail exactly and maybe even prove things mathematically about the system so that you know the limits and otherwise of the systems that you’re building. At that point I think all the world’s greatest minds should probably be thinking about this problem. So that was what I would be advocating to you know the Terence Tao’s of this world, the best mathematicians. Actually I’ve even talked to him about this—I know you’re working on the Riemann hypothesis or something which is the best thing in mathematics but actually this is more pressing. I have this sort of idea of like almost uh ‘Avengers assembled’ of the scientific world because that’s a bit of like my dream.
April 17th Stat News story: Influential Covid-19 model uses flawed methods and shouldn’t guide U.S. policies, critics say:
“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington.
Others experts, including some colleagues of the model-makers, are even harsher. “That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” said epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, home to several of the researchers who created the model, and who has served on a search committee for IHME. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”
Does Demis Hassabis think that coordinating pausing AI development globally is really plausible? If so, why/how/what’s the plan?
Or does he merely mean he thinks DeepMind could pause AI development at DeepMind and maybe should once we enter the “gray zone”/”middle zone” (the period before AGI when he says things will start “feeling interesting and strange”)?
Regarding this middle zone before AGI he says the signposts might be the AI “coming up with a truly original idea, creating something new, a new theory in science that ends up holding, maybe coming up with its own problem that it wants to solve.” But how sure is he that these signposts will happen before AGI, or long enough before that there’s still time to pause AI development and “prove things mathematically [...] so that you know the limits and otherwise of the systems that you’re building”?
If he wants to assemble all the world’s Terence Tao’s and top scientific minds to work on this because it’s a more pressing issue, then presumably that’s because he thinks there’s a lot of important work to be done, right? So then why not try to start all this work earlier rather than wait until things start feeling strange to him, like the systems almost have “sentience or awareness”? After all, he said AGI might be less than a decade away, so it’s not like we have forever. And maybe that way you can get the work done even if you don’t manage to persuade all the world’s top minds to come and work on it.
Demis Hassabis answered the question “Do you think DeepMind has a responsibility to hit pause at any point?” in 2022:
IMO The title is overly dramatic and seems to claim that the news about the new strain is more significant than I actually think it is in terms of how much it should cause us to update our views of what infection risk and US COVID-19 deaths will be in 2021.
FWIW when I first saw the title (on the EA Forum) my reaction was to interpret it with an implicit “[I think that] We must be very clear: fraud in the service of effective altruism is unacceptable”.
Things generally don’t just become true because people assert them to be—surely people on LW know that. I think habryka’s concern that not including “I think” in the title is a big deal is overblown. Dropping “I think” from the title is reasonable IMO to make the title more concise; I don’t anticipate it degrading the culture of LW. I also don’t see how it “bypasses epistemic defenses.” If the lack of inclusion of an “I think” in your title will worsen readers’ epistemics, then those readers seem to be at great risk of getting terrible epistemics from seeing any news headlines.
I don’t mean to say that there’s not value in using more nuanced language, including “I think” and similar qualifications to be more precise with ones words, just that I think the karma/vote ratio your post received is an over-reaction to concern about posts of your style degrading the level one “Attempt to describe the world accurately” culture of LW.
I made a visualization of Paul’s guesses to better understand how they overlap:
https://docs.google.com/spreadsheets/d/1x0I3rrxRtMFCd50SyraXFizSO-VRB3TrCRxUiWe5RMU/edit#gid=0