“A Muggle security expert would have called it fence-post security, like building a fence-post over a hundred metres high in the middle of the desert. Only a very obliging attacker would try to climb the fence-post. Anyone sensible would just walk around the fence-post, and making the fence-post even higher wouldn’t stop that.” —HPMOR, Ch. 115
(Not to be confused with the Trevor who works at Open Phil)
For those who might not have noticed, this actually is historic, they’re not just saying that- the top 350 people have effectively “come clean” about this, at once, in a schelling-point **kind-of** way.
The long years of staying quiet about this and avoiding telling other people your thoughts about AI potentially ending the world, because you’re worried that you’re crazy or that you take science fiction too seriously- those days **might have** just ended.
This was a credible signal, none of these 350 high-level people can go back and say “no, I never actually said that AI could cause extinction and AI safety should be a top global priority”, and from now on you and anyone else can cite this announcement to back up your views (instead of saying “Bill Gates, Elon Musk, and Stephen Hawking have all endorsed...”) and go straight to AI timelines (I like sending people Epoch’s Literature review).
EDIT: For the record, this might not be true, or it might not stick, and signatories retain ways of backing out or minimizing their past involvement. I do not endorse unilaterally turning this into more of a schelling point than it was originally intended to be.
Before I start criticizing, I would make it clear that I’m grateful for your work and I could not do better myself; I certainly did try, in fact I was one of the first in DC in 2018, but I could not do well since I was one of the many “socially-inept” people which are in fact a serious problem in DC (for the record: if you want to do AI policy, do not move to DC, first visit and have the people there judge your personality/charisma, the standards might be far higher or far lower than you might expect, they are now much better at testing people for fit than when I started 6 years ago).
I’m also grateful to see you put your work out there for review on Lesswrong, rather than staying quiet. I think the decision to attempt to be open vs. closed about AI policy work is vastly more complicated than most people in AI policy believe.
Your post is fantastic, especially the reflections.
You never mentioned the words “committee” or “chair” in this post. ?????????? Everything in Congress, other than elections and constituent calls, revolves around the congressional committees and particularly their chairmen. Is your model that congressional committees, just, aren’t important at all relative to party leadership in each chamber? If the balance of power has shifted that far by now, I wouldn’t know. Either way, congress is very much the kind of place where 20% of the congressmembers have >80% of the power, and the ones in the bottom 50% are the easiest to talk to, and their staffers exist to look important and talk to as many people as possible per day and make them feel heard, and their offices are focused on maintaining the appearance of being capable of substantially influencing legislation, in order to mitigate the risk that their voters and their professional network find out that they are in the bottom 50%. Over the centuries, Congress has become incredibly sophisticated at constructing mazes and leading people around. The Committee system is the first step to cutting through that and getting to where the bills are actually getting negotiated and written (primarily by lobbyists with de-facto personal ties to the office of the Chairs of the relevant committees and maybe the deputy chairs). Unless I’m wrong about this e.g. maybe a way larger share of policymaking power has accrued to the party leadership, who are even harder to meet with, or maybe the lobbyists from the big 5 tech companies are the main hotspot for tech-related policymaking in general including AI, and they meet with whoever they want, making the committee structure not very relevant to AI policy. It would have been great to hear more about the people you met at think tanks and the executive branch.
When it comes to foreign policy, which might be pretty important, a helpful way to look at it is like congress and other parliamentary bodies acting as a wall between domestic elites and the foreign policymaking institutions, and intelligence agencies as the main holders of real power. Obviously, these are people, and backdoor deals and revolving door employment is everywhere, so even this wall is fuzzy. But it is much more robust than, say, domestic policy (e.g. farm bills) where congress basically acts the conduit between elites and policy (e.g. like how most of the actual lawmaking work on capitol hill is done by lobbyists, not staffers). Intelligence agencies can easily bribe or infiltrate parliaments, and parliaments cannot easily bribe or infiltrate intelligence agencies. Authoritarian countries like China, on the other hand, don’t have real parliaments, and the strongman leader must mitigate the creep of rich domestic elites seeking policymaking influence (in reality it’s much more complex e.g. hybrid regimes, redirecting domestic elites to focus on local/provincial governments instead of the central/national government, etc your book might help with this but it’s important to note that books about intelligence agencies are products that need to optimize for entertainment in order to sell copies; books must be recommended by personal connections, and even then you never know, I might read and trust a book recommended to me by someone like Jason Matheny).
Lots of people asked me if I had draft legislation. Apparently, if you have regulatory ideas, people want to see that you have a (short) version of it written up like a bill.
They want you to propose solutions, they get annoyed when people come to them with a new issue they know nothing about and expect them to be the one to think of solutions. They want you to do the work writing up the final product and then hand it to them. If they have any issue with it, they’ll rewrite parts of it or throw it in the recycle bin.
In terms of my effect– I think I mostly just got them to think about it more and raised it in their internal “AI policy priorities” list. I think people forget that staffers have like 100 things on their priority list, so merely exposing and re-exposing them to these ideas can be helpful.
I’ve heard this characterized as “goldfish memory”. It’s important to note that many of the other 100 things on their priority list also have people trying to “expose and re-expose” them to ideas, and many staffers are hired for skill at pretending that they’re listening. I think you were correct to evaluate your work building relationships as more useful than this.
My experience in DC made me think that the Overton Window is extremely wide. Congress does not have cached takes on AI policy, and it seems like a lot of people genuinely want to learn. It’s unclear how long this will last (e.g., maybe AI risk ends up getting polarized), but we seem to be in a period of unusually high open-mindedness & curiosity.
I disagree that the Overton window in DC, or even Congress, is as wide as your impression. This is both for the reasons stated above, and because it seems very likely (>95%) that military-adjacent people in both the US and China are actively pursuing AI for things like economic growth/stabilization, military applications like EW and nuclear-armed cruise missiles, or for the data processing required for modern information warfare. I agree that we seem to be in a period of unusually high open-mindedness and curiosity.
With that said, I think coordination would be easier if people ended up being more explicit about what they believe, more explicit about specific policy goals they are hoping to achieve, and more explicit about their legible wins (and losses). In the absence of this, we run the risk of giving too much power and too many resources to people who “play the game”, develop influence, but don’t end up using their influence to achieve meaningful change.
I think that DC is a very Moloch-infested place, resulting in an intense and pervasive culture of nihilism- a near-universal belief that Moloch is inevitable. Prolonged exposure to that environment (several years), where everyone around you thinks this way, and will permanently mark you as low-social-status if you ever reveal you are one of those people with hope for the world, likely (>90%) has intense psychological effects on the AI Safety people in DC.
I think I would’ve written up a doc that explained my reasoning, documented the people I consulted with, documented the upside and downside risks I was aware of, and sent it out to some EAs.
internally screaming
I would’ve come with a printed-out 1-pager that explained what CAIS is & summarized the regulatory ideas in the NTIA response. I ended up doing this halfway through, and I would’ve done this sooner.
If you ever decide to write a doc properly explaining the situation with AI Safety to policymakers who read it, Scott Alexander’s Superintelligence FAQ is considered in high esteem, you could probably read it, think about how/why it was good at giving laymen a fair chance to understand the situation, and write a much shorter 1-pager yourself that’s optimized for the particular audience. I convinced both of my ~60-year-old parents to take AI safety seriously by asking them to read the AI chapter in Toby Ord’s The Precipice, so you might consider that instead.
Does anyone know what the base rate is for estranged family members making accusations against celebrity relatives? That’s a pretty important factor here e.g. it’s possible that journalists at reputable outlets are willing to write misleading stories about AI safety university groups because they have true statistics that they can cite (or use clever linguistic tricks and other tools of the trade to straight-up lie about those statistics in plausibly deniable ways, which sadly also still happens even at the most reputable outlets), but can’t write honest stories about accusations from estranged family members because of journalistic ethics.
Or maybe editors at news outlets and other varieties of corporate executives all have estranged family members so there’s a norm against it, which sometimes holds and sometimes doesn’t. All of it centers around what the base rate is, a single number, which I don’t know. But it’s impossible to investigate this topic in a truthseeking way and simultaneously not attempt to find the number that all the other calculations indisputably revolve around. The base rate of false rape accusations for normal people is incredibly low, likely because the victim loses social status in an extreme way (some right-wingers claim that accusers gain social status and this claim has been thoroughly researched and discredited) and the base rate of non-reporting is very high, and both of those numbers are absolutely critical for even beginning to understand the problem. The same goes here.
The only reason that someone like Cade Metz is able to do what he does, performing at the level he has been, with a mind like what he has, is because people keep going and talking to him. For example, he might not even have known about the “among the doomsayers” article until you told him about it (or found out about it much sooner).
I can visibly see you training him, via verbal conversation, how to outperform the vast majority of journalists at talking about epistemics. You seemed to stop towards the end, but Metz nonetheless probably emerged from the conversation much better prepared to think up attempts to dishonestly angle-shoot the entire AI safety scene, as he has continued to do over the last several months.
Rationalists are, in Scott Alexander’s formulation, missing a mood, or rather, they are drawn from a pool of mostly men who are missing one. “Normal” people instinctively grasp social norms without having them explained. Rationalists lack this instinct.
In particular, they struggle with small talk and other social norms around speech, because they naively think words are a vehicle for their literal meanings. Yud’s sequences help this by formalizing the implicit decisions that normal people make.
...
The quokka, like the rationalist, is a creature marked by profound innocence. The quokka can’t imagine you might eat it, and the rationalist can’t imagine you might deceive him. As long they stay on their islands, they survive, but both species have problems if a human shows up.
In theory, rationalists like game theory, in practice, they need to adjust their priors. Real-life exchanges can be modeled as a prisoner’s dilemma. In the classic version, the prisoners can’t communicate, so they have to guess whether the other player will defect or cooperate.
The game changes when we realize that life is not a single dilemma, but a series of them, and that we can remember the behavior of other agents. Now we need to cooperate, and the best strategy is “tit for two tats”, wherein we cooperate until our opponent defects twice.
The problem is, this is where rationalists hit a mental stop sign. Because in the real world, there is one more strategy that the game doesn’t model: lying. See, the real best strategy is “be good at lying so that you always convince your opponent to cooperate, then defect”.
And rationalists, bless their hearts, are REALLY easy to lie to. It’s not like taking candy from a baby; babies actually try to hang onto their candy. The rationalists just limply let go and mutter, “I notice I am confused”.
...
Rationalists = quokkas, this explains a lot about them. Their fear instincts have atrophied. When a quokka sees a predator, he walks right up; when a rationalist talks about human biodiversity on a blog under almost his real name, he doesn’t flinch away.
A normal person learns from social cues that certain topics are forbidden, and that if you ask questions about them, you had better get the right answer, which is not the one with the highest probability of being true, but the one with the highest probability of keeping your job.
This ability to ask uncomfortable questions is one of the rationalist’s best and worst attributes, because mental stop signs, like road stop signs, actually exist to keep you safe, and although there may be times one should disregard them, most people should mostly obey them,
...
Apropos of the game theory discussion above, if there is ONE thing I can teach you with this account, it’s that you have evolved to be a liar. Lying is “killer app” of animal intelligence, it’s the driver of the arms race that causes intelligence to evolve.
...
The main way that you stop being a quokka is that you realize there are people in the world who really want to hurt you. There are people who will always defect, people whose good will is fake, whose behavior will not change if they hear the good news of reciprocity.
So things that everyone warns you not to do, like going and talking to people like Cade Metz, might seem like a source of alpha, undersupplied by the market. But in reality there is a good reason why everyone at least tried to coordinate not to do it, and at least tried to make it legible why people should not do that. Here the glass has already been blown into a specific shape and cooled.
Do not talk to journalists without asking for help. You have no idea how much there is to lose, even just from a short harmless-seeming conversation where they are able to look at how your face changes as you talk about some topics and avoid others.
Human genetic diversity implies that there are virtually always people out there who are much better at that than you’d expect from your own life experience of looking at people’s facial expressions, no matter your skill level, and other factors indicate that these people probably started pursuing high-status positions a long time ago.
For those who might not have noticed, this actually is historic, they’re not just saying that- the top 350 people have effectively “come clean” about this, at once, in a schelling-point **kind-of** way.
The long years of staying quiet about this and avoiding telling other people your thoughts about AI potentially ending the world, because you’re worried that you’re crazy or that you take science fiction too seriously- those days **might have** just ended.
This was a credible signal, none of these 350 high-level people can go back and say “no, I never actually said that AI could cause extinction and AI safety should be a top global priority”, and from now on you and anyone else can cite this announcement to back up your views (instead of saying “Bill Gates, Elon Musk, and Stephen Hawking have all endorsed...”) and go straight to AI timelines (I like sending people Epoch’s Literature review).
EDIT: For the record, this might not be true, or it might not stick, and signatories retain ways of backing out or minimizing their past involvement. I do not endorse unilaterally turning this into more of a schelling point than it was originally intended to be.