You (Yes, You) Need A February 2020 Checklist for AI Policy
TL;DR: You (Yes You) should prepare for a “February 2020” moment where suddenly AI policy becomes the most important issue in the world. You should be ready to take action if and when it does, in a detailed way.
(Epistemic status: originally written for an event in early 2026; have heard from some folks that they found planning processes inspired by this memo very helpful for the smaller-scale OpenAI / Hugging Face response, so very quickly redacting a few things and posting this as-is.)
Many people in the AI policy space assume that eventually we’ll be at an Overton Window-shifting crisis moment, that opens the floodgates for the really good policies all along that we had.
But when you look at successful handling of crisis moments, there was no time to think – people applied strategies they’d learned via academic study or previous professional work, and then moved against them rapidly. For example, after 9/11, the US government operationalized past reports on intelligence and law enforcement reform and institutionalized them into law (good?)[1] and also picked an enemy to fight based on past history, Iraq (bad). Or in the 2008 financial crisis, Ben Bernanke brought deep academic experience studying financial panics and the Great Depression, partnered with Tim Geithner and Henry Paulson’s market and policy experience and deep networks. Or in 2020, Anthony Fauci essentially cashed in 30 years [2]of epidemic-fighting expertise and relationships in one go (whether well or poorly is outside the scope of this piece, but know that I Have Feelings).
I assert that the term “crisis” tends to include two different modes of American security policy
Ordinary mode, where things move slowly and deliberately; there are crises, but they’re manageable and embedded in environments with substantial reserves of capacity to address them – Cold War standoffs; the 2000 recession; a bad flu season.
Extraordinary mode, where we’re playing utter Calvinball, no one sleeps for a year, and all the rules get made up along the way – the day after Pearl Harbor; September 12th, 2001; the 2008 financial crisis; the onset of COVID in 2020.
This memo means the latter.
Other folks propose great timeline uncertainty, and note the benefits of long timelines for getting it right. This community broadly, and perhaps you personally, Gentle Reader, should absolutely put bets behind longer timelines. I like longer timelines. Longer timelines have among other advantages, gaining approximately 8.3 billion person-years of human survival per additional year of wall-clock time, which sounds great. Those still probably end up in extraordinary mode, but you have plausibly more expected months of ordinary mode before you get there.
But I also think you should take very short timelines very seriously, and specifically build a plan against it. [3] Anthropic thinks we have ~2 to 3 years to ~AGI, and they sure do seem to mean it. And frankly, when I do my every ~3 months trip into the Bay Area, many people (including some reading this who don’t work at Anthropic) sure do seem to talk as if they expect things to get very very very very crazy in the next 2-3 years, not 5-10.
But when I ask those folks what their plan for extraordinary mode, for seizing that Overton-window-shifting moment when it comes, they don’t have a specific answer. It’s some combo of an off-the-cuff, “I don’t know, I guess fly to DC and try to talk to everyone I have relationships with there, try to advocate for good stuff.”
(This criticism also applies to me; I haven’t built this plan fully yet for my own job. Working on it now.)
There’s no fucking time to think when policymakers are in extraordinary mode.
Every AI org should have a detailed runbook, with specific tasks, updated twice a year, [4]against whatever you think your 1 to 3 most likely “AI has suddenly become the top issue in American life” crises are. (You’ll probably get this guess wrong, but an imperfect plan is better than no plan). This will probably change as American politics changes. You should have the ability to immediately just start executing on as much as possible if you think the crisis is a reasonable chance of hitting in some time in the next one to two years.
(Note the obvious risk: if you’re wrong, you’re really wrong, and you poison the well for the rest of us. I suspect, therefore, that you won’t be going to this checklist until after you ideally should have, because of these social pressures. So don’t worry about that too much.)
Ideally, this checklist process should be iterative, and you should reason backwards from the checklist to identify gaps in your current efforts. Specifically, you should ask yourself the question: “What do I really wish I had done by then, so that I’m ready?” In my example checklist below, you’ll note that it works best if you’ve spent time building relationships with key stakeholders, and their staffers, and their staffers’ staffers, so that the people who are in the White House Situation Room and the relevant Congressional committees make the right calls. Similarly, the hard work of policy still needs to be done, both in terms of preparing (and ideally, getting implemented) policy ideas in advance.
Acknowledgements: My thanks to my work colleagues, especially Jeffrey Ladish and Eli Tyre, for pushing my thinking on this. They don’t necessarily endorse any of this.
- ^
I recommend the book “Blinking Red,” by Michael Allen, in this regard: https://amzn.to/3NYKfLA
- ^
Fauci had been in his leadership role at NIH – a civil service role, so no one could kick him out of it – so long that George Bush Senior praised him in the 1988 Presidential Campaign: https://www.c-span.org/clip/public-affairs-event/user-clip-bush-mentions-fauci/4875658
- ^
My shoulder Eliezer and Nate say, “you just die,” and sure, that’s possible. But unfortunately, I’m the kind of stubborn where even if I think I’m gonna lose, I’ll fight.
- ^
Ideally more often, but I don’t think this is going to happen in real life until we internalize the idea that we can have our in-house LLM accounts read all of our emails and texts, identify things we should update, and update the run books dynamically.
One scenario I’ve thought of is the Hackening, i.e. in 3-12 months, someone releases an open source model equivalent to Mythos, different from previous models for having the Juice (autonomous end-to-end exploit construction), and the world is inadequately prepared for the mass of newly minted or empowered hackers. Many would be individuals that don’t care about causing international incidents if it gives short-term gain, and all of them can now probe the world’s entire attack surface. Even with efforts like Glasswing, there’s still likely to be a very long tail of targets that don’t have defense budgets or prepared SOCs. I haven’t seen anyone try to model this in detail, but what comes to my mind is that there’d be a gold rush of attacks while systems are still undefended, and enough ransomware/credential harvests/blackmails/lockouts could happen to trigger a recession. Attribution could be hard enough, and enough states tempted to join the chaos, that this causes heightened international tensions.
This seems like the sort of wake-up call that if nothing else of similar magnitude happened by then, it would be what causes AI policy to go mainstream. I don’t know what the default crisis response to that looks like, or what improvements on it could be. There might also be moderating factors, like control of compute, offense/defense asymmetries I didn’t think of, the economics of running attacks, or other things, that could reduce the severity of the scenario. It’s also possible that someone manages to make a harness that has the Juice even when a model in a simple agentic harness doesn’t.
One likely default response is I expect people to start clamoring hard for open-source bans, and depending on what happens in November 2026, they could plausibly succeed.
More generally, I expect the public to be even more anti-AI, and for politicians to sate that appetite by starting to ban open-source.
One good example of this is soft law/FUD from the federal administration:
Or even just floating very interventionist ideas:
This is not that bad, but it is a problem because it crowds out misalignment concerns and also because the path the US takes is not likely to be one that favors slowing down/pausing, because the open-source AIs won’t be misaligned (most likely.), and the people in power will just see it as a security problem.
You know, I never thought about it, but it does seem possible that until the dangerous open source model is distilled, the US (and other countries) could actually proactively lean on global compute providers to not host it, driving the market price way up, if/when LLMjacking rates fall.
I wonder how long people in power will continue to see examples of misalignment and just think of it as a product or security problem? Would there be something that would nudge them towards thinking of it as a moral/ethics problem?
Great post, excepct, maybe now is the time to be going out and pushing for policies?
I would not disagree with this being a good time for advocacy! I suspect we’ll have at least one bigger shock still, and planning accordingly seems worthwhile. Walk and chew gum, etc.
The point of the post is not that we shouldn’t be doing everything we can to stop a potential “epidemic”. It’s that we should have a plan for what to do when an “epidemic” happens anyways and there is enormous amounts of political capital suddenly available to get things done.
The point is that when or if that fire hose ever opens we should have a plan to point it at the fire because we won’t have time to aim. We could very easily point the fire house in the wrong direction as has occurred often in history.
“When or if”—what if it is open now
I think we can probably do more now than most people think—or at least we could if we had prepared better. But no, this is not what ‘open fire hose’ looks like. Frankly I’m not sure I’ve ever seen an open fire hose moment (in the US) in my lifetime. In 2008 the economists were saying how many times bigger the stimulus needed to be, and we didn’t do it, and slogged through a decade-long slow recovery. In 2020 we removed some but not all of the restrictions slowing down covid vaccine development and definitely didn’t make what would have been some relatively low-cost low-risk high-impact moves, and made no moves to change policy in obvious good ways moving forward.
An open fire hose moment, at least to me, looks like the market crash in 1929 (3.5 years(!) before the new deal), Poland in 1939, Pearl Harbor in 1942, or maybe Sputnik in 1957. Right now (if my quick estimation is even close) private investment in AI in one year is larger in inflation adjusted terms, both in dollars and as a % of GDP, than the entire Apollo program was. IMO a firehose moment would at minimum include recognizing that the risks are at least USSR-level and deserve being willing to mobilize at Cold War scale in response for as long as it takes.
But if I’m wrong, and there is an open firehose moment now, then there’s a lot of things we should and in principle could immediately do. Top of mind:
Lock down chip exports much tighter—net negative cost, direct economic benefits to the US
Provide extensive support to AI labs to beef up their physical security and cybersecurity
Create a dedicated civilian regulatory agency with the authority to (dual mandate!) promote the safe advancement of AI
Implement large scale infrastructure upgrades—physical and digital grid hardening, renewable energy production, large scale widespread energy storage, water treatment and water recycling capacity. This may not sound like an AI measure, but resilience against shocks is always valuable in a crisis, many types of attacks or disasters are possible and these are good on the merits even if no specific AI-driven crisis occurs. Trillions of dollars needed, minimum
Implement tight controls in biotech to ensure labs that make DNA/RNA/proteins/etc. are screening all orders for dangerous substances or evidence of nefarious intent (with humans and with the best existing AI)
Implement society-wide preparedness measures against future (engineered or natural) pandemic threats—air filtration, UVC, PPE stockpiles, vaccine production capacity, FDA (or local equivalent) reforms
Reform land use and energy market regulations nationwide to make building infrastructure (and industrial capacity, and housing, why not) faster, cheaper, and smoother—keep more data centers domestic, and encourage using AI investment to simultaneously make US energy and water systems cleaner and more modern and better—this also cushions against the ‘bubble’ narrative because these kinds of projects have value with or without AI if you do them right
Encourage development and deployment of robotics for civilian benefit
Educate the public about how the economy even works, including why automation can and should be net beneficial with the right systems in place
The time is now. OpenAI has stopped training.
It certainly “feels” like January 2020...
Applauding five things that are crucial to act on now, that this post does well:
Highlight the need to establish a plan rationally before a crisis mode and tension make rationality difficult
Illustrate (with personal commitment) that we can each take steps to prepare our communities/organizations
Encourage setting up relationships for crisis management
Elevate policy as a lever to manage the crisis
Name the risk that “ordinary” overton dynamics are holding us back from seriously considering actions that could avert the crisis
From personal experience with misrepresentations over the last 5 years, I will also add my personal warning that participants in many, many conversations and comments have personal or parochial incentives and “risks” (to their interests), where alignment requires deliberate effort lest they take precedence over the “risks” under discussion.
In other words: unity is essential
That is, I need to prepare for a policy from anti-AI activists which will be created in haste, implemented by bureaucrats, justified using exaggerated claims, done in arbitrary ways which make no sense even compared to the policy’s stated goals; which restricts the rights of billions of people, causes enormous economic damage, and has little effect on casualties anyway, while politicians and well connected people just don’t bother obeying the restrictions themselves?
I think the lesson to be learned from the Covid reaction is not the lesson you took from it.
The lesson he seems to be taking from the COVID reaction is that people do stupid things when there’s a crisis, so you should have a clear plan prepared to help them do non-stupid things. I don’t see any advocacy for maximum panic here.
The other lesson from COVID is that people go from not caring at all to caring a whole bunch right on a dime when they become actually worried that something could affect them personally in a pretty dramatic way—and more importantly, when others around them start publicly expressing worry, there’s a rapid collective sea change.
This happens because humans tend strongly to do their epistemics as groups, not individuals.
I believe Jiro’s objection is that he expects ~anything done on the matter to be stupid, and he would prefer that people simply not take action, that there be a clear plan prepared to help them not do things.
People are downvoting you without necessarily knowing where the crux is and that’s a little unfair.
Is it that
1) Strange things won’t happen with AI?
2) There doesn’t need to be any action taken by governments if strange things happen
3) Even if strange things happen (or are anticipated to happen) and government responds, it’ll be ineffectual and make a bad problem worse?
(3 and 2 are slightly different. For example, you hold (2) if you believe corporate governance is sufficient for any AI strangeness, if any.)
For me, a big part of the reason is that it went off topic immediately in an area that’s still charged by politics, so I didn’t want the discussion to centralize around something that would be predictably useless but very dramatic/distracting.
In essence, I’m trying to cut short a likely demon thread around COVID.