Risk of misuse
The thought had never previously struck us.
This seems like one of the most tractable things to address to reduce AI risk.
If 5 years from now anyone developing AI or biotechnology is still not thinking (early and seriously) about ways their work could cause harm that other people have been talking about for years, I think we should consider ourselves to have failed.
The David Silver on it being okay to have AGIs with different goals part worried me because it sounded like he wasn’t at all thinking about the risk from misaligned AI. It seemed like he was saying we should create general intelligence regardless of its goals and values, just because it’s intelligent.
In this post Zvi doesn’t try to forecast how many infections/cases/deaths there would be in the US without this new strain (unless I missed it… it is a long post). Yet he really should, because doing so will lead one to realize that the US is likely going to be at or close to herd immunity by ~May-June anyway, so a new transmissible strain that becomes dominant in the US around that same period can’t plausibly make as huge of a difference as Zvi seems to be saying in this post.
Good Judgment’s median estimate for “How many total cases of COVID-19 in the U.S. will be estimated as of 31 March 2021?” is ~130M currently. And Good Judgment’s median estimate for “When will enough doses of FDA-approved COVID-19 vaccine(s) to inoculate 100 million people be distributed in the United States?” is ~May 1st currently. https://goodjudgment.io/covid/dashboard/
Assuming that 20% of vaccines go to people who had already been infected, this would mean that by May, approximately ~220M people (220M = ~140M + 0.8*100M) the US will be immune, or about 66% of the population. This could easily be higher or lower, but the point is that we’re going to be at or close to herd immunity by the time Zvi says this new viral strain would start becoming dominant in the US.
In short, the news would be much worse if this new viral strain had spread to the degree that it has now several months ago. But in reality, I think we’ll be at or close to herd immunity already by the time it becomes prominent, so it won’t make that much of a difference.
EDIT: I misread Zvi’s piece initially and mistakenly thought he wrote that that the new strain wouldn’t become dominant in the US until May. I now see that he says “Instead of that being the final peak and things only improving after that, we now face a potential fourth wave, likely cresting between March and May, that could be sufficiently powerful to substantially overshoot herd immunity.” Taking this view as true instead makes me see the new strain as significantly worse news: specifically, this two-month shift might be sufficient to make an additional ~10-15% of the population get infected/sick before herd immunity is reached. (I still think the post title is overblown, but still this is a significant update for me.)
On the subject of DeemMind and pausing AI development, I’d like to highlight Demis Hassabis’s remark on this topic in a DeepMind podcast interview a year ago:
‘Avengers assembled’ for AI Safety: Pause AI development to prove things mathematically
Hannah Fry (17:07):
You said you’ve got this sort of 20-year prediction and then simultaneously where society is in terms of understanding and grappling with these ideas. Do you think that DeepMind has a responsibility to hit pause at any point?
Demis Hassabis (17:24):
Potentially. I always imagine that as we got closer to the sort of gray zone that you were talking about earlier, the best thing to do might be to pause the pushing of the performance of these systems so that you can analyze down to minute detail exactly and maybe even prove things mathematically about the system so that you know the limits and otherwise of the systems that you’re building. At that point I think all the world’s greatest minds should probably be thinking about this problem. So that was what I would be advocating to you know the Terence Tao’s of this world, the best mathematicians. Actually I’ve even talked to him about this—I know you’re working on the Riemann hypothesis or something which is the best thing in mathematics but actually this is more pressing. I have this sort of idea of like almost uh ‘Avengers assembled’ of the scientific world because that’s a bit of like my dream.
To add some more emphasis to my point, because I think it deserves more emphasis:
Quoting the interview Jacy linked to:
Your paper also says that “[w]ithout being overly alarmist, this should serve as a wake-up call for our colleagues” — what is it that you want your colleagues to wake up to? And what do you think that being overly alarmist would look like?
We just want more researchers to acknowledge and be aware of potential misuse. When you start working in the chemistry space, you do get informed about misuse of chemistry, and you’re sort of responsible for making sure you avoid that as much as possible. In machine learning, there’s nothing of the sort. There’s no guidance on misuse of the technology.
I know I’m not saying anything new here, and I’m merely a layperson without ability to verify the truth of the claim I highlighted in bold above, but I do want to emphasize further:
I seems clear that changing the machine learning space so that it is like the chemistry space in the sense that you do get informed about ways machine learning can be misused and cause harm, is something that we should all push to make happen as soon as we can. (Also expanding the discussion of potential harm beyond harm caused from misuse to any harm related to the technology.)
Years ago I recall hearing Stuart Russell mention the analogy of how civil engineers don’t have a separate field for bridge safety; rather bridge safety is something all bridge designers are educated on and concerned about, and he similarly doesn’t want the field of AI safety to be separate from AI but wants all people working on AI to be educated on and concerned with risks from AI.
This is the same point I’m saying here, and I’m saying it again because it seems like the present machine learning space is still far from this point and we as a community really do need to devote more efforts to ensuring that we change this in the near future.
In a March 15th episode of the DeepMind podcast, Demis Hassabis said he has talked to Terence Tao about working on AI safety:
I always imagine that as we got closer to the sort of gray zone that you were talking about earlier, the best thing to do might be to pause the pushing of the performance of these systems so that you can analyze down to minute detail exactly and maybe even prove things mathematically about the system so that you know the limits and otherwise of the systems that you’re building. At that point I think all the world’s greatest minds should probably be thinking about this problem. So that was what I would be advocating to you know the Terence Tao’s of this world, the best mathematicians. Actually I’ve even talked to him about this—I know you’re working on the Riemann hypothesis or something which is the best thing in mathematics but actually this is more pressing. I have this sort of idea of like almost uh ‘Avengers assembled’ of the scientific world because that’s a bit of like my dream.
If we interpret his claim as an additional 0.5% wealth tax do you still think he is overstating his claim?
It’d be nice if Hassabis made another public statement about his views on pausing AI development and thoughts on the FLI petition. If now’s not the right time in his view, when is? And what can he do to help with coordination of the industry?
You’re probably already aware of this, but just in case not:
Demis Hassabis said the following about getting Terrence Tao to work on AI safety:
I always imagine that as we got closer to the sort of gray zone that you were talking about earlier, the best thing to do might be to pause the pushing of the performance of these systems so that you can analyze down to minute detail exactly and maybe even prove things mathematically about the system so that you know the limits and otherwise of the systems that you’re building. At that point I think all the world’s greatest minds should probably be thinking about this problem. So that was what I would be advocating to you know the Terence Tao’s of this world, the best mathematicians. Actually I’ve even talked to him about this—I know you’re working on the Riemann hypothesis or something which is the best thing in mathematics but actually this is more pressing. I have this sort of idea of like almost uh ‘Avengers assembled’ of the scientific world because that’s a bit of like my dream.
April 17th Stat News story: Influential Covid-19 model uses flawed methods and shouldn’t guide U.S. policies, critics say:
“It’s not a model that most of us in the infectious disease epidemiology field think is well suited” to projecting Covid-19 deaths, epidemiologist Marc Lipsitch of the Harvard T.H. Chan School of Public Health told reporters this week, referring to projections by the Institute for Health Metrics and Evaluation at the University of Washington.
Others experts, including some colleagues of the model-makers, are even harsher. “That the IHME model keeps changing is evidence of its lack of reliability as a predictive tool,” said epidemiologist Ruth Etzioni of the Fred Hutchinson Cancer Center, home to several of the researchers who created the model, and who has served on a search committee for IHME. “That it is being used for policy decisions and its results interpreted wrongly is a travesty unfolding before our eyes.”
Does Demis Hassabis think that coordinating pausing AI development globally is really plausible? If so, why/how/what’s the plan?
Or does he merely mean he thinks DeepMind could pause AI development at DeepMind and maybe should once we enter the “gray zone”/”middle zone” (the period before AGI when he says things will start “feeling interesting and strange”)?
Regarding this middle zone before AGI he says the signposts might be the AI “coming up with a truly original idea, creating something new, a new theory in science that ends up holding, maybe coming up with its own problem that it wants to solve.” But how sure is he that these signposts will happen before AGI, or long enough before that there’s still time to pause AI development and “prove things mathematically [...] so that you know the limits and otherwise of the systems that you’re building”?
If he wants to assemble all the world’s Terence Tao’s and top scientific minds to work on this because it’s a more pressing issue, then presumably that’s because he thinks there’s a lot of important work to be done, right? So then why not try to start all this work earlier rather than wait until things start feeling strange to him, like the systems almost have “sentience or awareness”? After all, he said AGI might be less than a decade away, so it’s not like we have forever. And maybe that way you can get the work done even if you don’t manage to persuade all the world’s top minds to come and work on it.
Demis Hassabis answered the question “Do you think DeepMind has a responsibility to hit pause at any point?” in 2022:
IMO The title is overly dramatic and seems to claim that the news about the new strain is more significant than I actually think it is in terms of how much it should cause us to update our views of what infection risk and US COVID-19 deaths will be in 2021.
FWIW when I first saw the title (on the EA Forum) my reaction was to interpret it with an implicit “[I think that] We must be very clear: fraud in the service of effective altruism is unacceptable”.
Things generally don’t just become true because people assert them to be—surely people on LW know that. I think habryka’s concern that not including “I think” in the title is a big deal is overblown. Dropping “I think” from the title is reasonable IMO to make the title more concise; I don’t anticipate it degrading the culture of LW. I also don’t see how it “bypasses epistemic defenses.” If the lack of inclusion of an “I think” in your title will worsen readers’ epistemics, then those readers seem to be at great risk of getting terrible epistemics from seeing any news headlines.
I don’t mean to say that there’s not value in using more nuanced language, including “I think” and similar qualifications to be more precise with ones words, just that I think the karma/vote ratio your post received is an over-reaction to concern about posts of your style degrading the level one “Attempt to describe the world accurately” culture of LW.
It seems troubling that one of the most upvoted COVID-19 post on LessWrong is one that argued for a prediction that I think we should score really poorly.
I agree. FWIW, I strong-downvoted this post in December. I think this is the first LW post that I have strong-downvoted before.
Additionally, I commented on it (and on threads where this post was shared elsewhere, e.g. on Facebook) to explain my disagreement with it, and recorded ~the lowest forecast of anyone who submitted their forecast here that there’d be a 4th wave in the US in 2021.
What I failed to do was offer to bet here on the 4th wave question. I think the only time that I tried to make a bet on this topic was in a discussion on Facebook (set to friends-only) that began with “Well, it’s time to pull the fire alarm on the UK mutation.” I commented on the post on 12/26/20 with the following:
Would you be interested in operationalizing a bet on this? (If you don’t think its good practice to bet money on COVID infections/cases/deaths or otherwise aren’t interested in betting money, we can just make it a reputational bet.)
I get the sense that like Zvi you are being too pessimistic about how bad the new strain will be for the US relative to how bad thing would have been even without the new strain.
However, the bet never came to fruition.
The sample Nate linked is 30 pages and 12,267 words. So that works out to ~730 pages for a run.
$20,000/300,000 words = $1 per 15 words. If an author writing it manually could average 15 wpm, that would be $60/hour.
For another worthwhile free money opportunity available until December 25th, see: Make a $10 donation into $35
Also every one of the organizations you named is a capabilities company which brands itself based on the small team they have working on alignment off on the side.
I’m not sure whether OpenAI was one of the organizations named, but if so, this reminded me of something Scott Aaronson said on this topic in the Q&A of his recent talk “Scott Aaronson Talks AI Safety”:
Maybe the one useful thing I can say is that, in my experience, which is admittedly very limited—working at OpenAI for all of five months—I’ve found my colleagues there to be extremely serious about safety, bordering on obsessive. They talk about it constantly. They actually have an unusual structure, where they’re a for-profit company that’s controlled by a nonprofit foundation, which is at least formally empowered to come in and hit the brakes if needed. OpenAI also has a charter that contains some striking clauses, especially the following:
We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project.
Of course, the fact that they’ve put a great deal of thought into this doesn’t mean that they’re going to get it right! But if you ask me: would I rather that it be OpenAI in the lead right now or the Chinese government? Or, if it’s going to be a company, would I rather it be one with a charter like the above, or a charter of “maximize clicks and ad revenue”? I suppose I do lean a certain way.
Source: 1:12:52 in the video, edited transcript provided by Scott on his blog.
In short, it seems to me that Scott would not have pushed back on a claim that OpenAI is an organization” that seem[s] like the AI research they’re doing is safety research” in the way you did Jim.
I assume that all the sad-reactions are sadness that all these people at the EAGx conference aren’t noticing that their work/organization seems bad for the world on their own and that these conversations are therefore necessary. (The shear number of conversations like this you’re having also suggests that it’s a hopeless uphill battle, which is sad.)
So I wanted to bring up what Scott Aaronson said here to highlight that “systemic change” interventions are necessary also. Scott’s views are influential; potentially targeting talking to him and other “thought leaders” who aren’t sufficiently concerned about slowing down capabilities progress (or who don’t seem to emphasize enough concern for this when talking about organizations like OpenAI) would be helpful, of even necessary, for us to get to a world a few years from now where everyone studying ML or working on AI capabilities is at least aware of arguments about AI alignment and why increasing increasing AI capabilities seems harmful.
How many years do you think it will be until we see (in public) an agent which only gets screen pixels as input, has human-level apm and reaction speed, and is very clearly better than the best humans?
Respondents had a median prediction of two years and an expertise-weighted mean prediction of a little less than four years.
It’s now been about two years and this hasn’t happened yet. It seems like that might just be the case because DeepMind stopped work on this?
I made a visualization of Paul’s guesses to better understand how they overlap:
https://docs.google.com/spreadsheets/d/1x0I3rrxRtMFCd50SyraXFizSO-VRB3TrCRxUiWe5RMU/edit#gid=0