Before I start criticizing, I would make it clear that I’m grateful for your work and I could not do better myself; I certainly did try, in fact I was one of the first in DC in 2018, but I could not do well since I was one of the many “socially-inept” people which are in fact a serious problem in DC (for the record: if you want to do AI policy, do not move to DC, first visit and have the people there judge your personality/charisma, the standards might be far higher or far lower than you might expect, they are now much better at testing people for fit than when I started 6 years ago).
I’m also grateful to see you put your work out there for review on Lesswrong, rather than staying quiet. I think the decision to attempt to be open vs. closed about AI policy work is vastly more complicated than most people in AI policy believe.
Your post is fantastic, especially the reflections.
You never mentioned the words “committee” or “chair” in this post. ?????????? Everything in Congress, other than elections and constituent calls, revolves around the congressional committees and particularly their chairmen. Is your model that congressional committees, just, aren’t important at all relative to party leadership in each chamber? If the balance of power has shifted that far by now, I wouldn’t know. Either way, congress is very much the kind of place where 20% of the congressmembers have >80% of the power, and the ones in the bottom 50% are the easiest to talk to, and their staffers exist to look important and talk to as many people as possible per day and make them feel heard, and their offices are focused on maintaining the appearance of being capable of substantially influencing legislation, in order to mitigate the risk that their voters and their professional network find out that they are in the bottom 50%. Over the centuries, Congress has become incredibly sophisticated at constructing mazes and leading people around. The Committee system is the first step to cutting through that and getting to where the bills are actually getting negotiated and written (primarily by lobbyists with de-facto personal ties to the office of the Chairs of the relevant committees and maybe the deputy chairs). Unless I’m wrong about this e.g. maybe a way larger share of policymaking power has accrued to the party leadership, who are even harder to meet with, or maybe the lobbyists from the big 5 tech companies are the main hotspot for tech-related policymaking in general including AI, and they meet with whoever they want, making the committee structure not very relevant to AI policy. It would have been great to hear more about the people you met at think tanks and the executive branch.
When it comes to foreign policy, which might be pretty important, a helpful way to look at it is like congress and other parliamentary bodies acting as a wall between domestic elites and the foreign policymaking institutions, and intelligence agencies as the main holders of real power. Obviously, these are people, and backdoor deals and revolving door employment is everywhere, so even this wall is fuzzy. But it is much more robust than, say, domestic policy (e.g. farm bills) where congress basically acts the conduit between elites and policy (e.g. like how most of the actual lawmaking work on capitol hill is done by lobbyists, not staffers). Intelligence agencies can easily bribe or infiltrate parliaments, and parliaments cannot easily bribe or infiltrate intelligence agencies. Authoritarian countries like China, on the other hand, don’t have real parliaments, and the strongman leader must mitigate the creep of rich domestic elites seeking policymaking influence (in reality it’s much more complex e.g. hybrid regimes, redirecting domestic elites to focus on local/provincial governments instead of the central/national government, etc your book might help with this but it’s important to note that books about intelligence agencies are products that need to optimize for entertainment in order to sell copies; books must be recommended by personal connections, and even then you never know, I might read and trust a book recommended to me by someone like Jason Matheny).
Lots of people asked me if I had draft legislation. Apparently, if you have regulatory ideas, people want to see that you have a (short) version of it written up like a bill.
They want you to propose solutions, they get annoyed when people come to them with a new issue they know nothing about and expect them to be the one to think of solutions. They want you to do the work writing up the final product and then hand it to them. If they have any issue with it, they’ll rewrite parts of it or throw it in the recycle bin.
In terms of my effect– I think I mostly just got them to think about it more and raised it in their internal “AI policy priorities” list. I think people forget that staffers have like 100 things on their priority list, so merely exposing and re-exposing them to these ideas can be helpful.
I’ve heard this characterized as “goldfish memory”. It’s important to note that many of the other 100 things on their priority list also have people trying to “expose and re-expose” them to ideas, and many staffers are hired for skill at pretending that they’re listening. I think you were correct to evaluate your work building relationships as more useful than this.
My experience in DC made me think that the Overton Window is extremely wide. Congress does not have cached takes on AI policy, and it seems like a lot of people genuinely want to learn. It’s unclear how long this will last (e.g., maybe AI risk ends up getting polarized), but we seem to be in a period of unusually high open-mindedness & curiosity.
I disagree that the Overton window in DC, or even Congress, is as wide as your impression. This is both for the reasons stated above, and because it seems very likely (>95%) that military-adjacent people in both the US and China are actively pursuing AI for things like economic growth/stabilization, military applications like EW and nuclear-armed cruise missiles, or for the data processing required for modern information warfare. I agree that we seem to be in a period of unusually high open-mindedness and curiosity.
With that said, I think coordination would be easier if people ended up being more explicit about what they believe, more explicit about specific policy goals they are hoping to achieve, and more explicit about their legible wins (and losses). In the absence of this, we run the risk of giving too much power and too many resources to people who “play the game”, develop influence, but don’t end up using their influence to achieve meaningful change.
I think that DC is a very Moloch-infested place, resulting in an intense and pervasive culture of nihilism- a near-universal belief that Moloch is inevitable. Prolonged exposure to that environment (several years), where everyone around you thinks this way, and will permanently mark you as low-social-status if you ever reveal you are one of those people with hope for the world, likely (>90%) has intense psychological effects on the AI Safety people in DC.
I think I would’ve written up a doc that explained my reasoning, documented the people I consulted with, documented the upside and downside risks I was aware of, and sent it out to some EAs.
internally screaming
I would’ve come with a printed-out 1-pager that explained what CAIS is & summarized the regulatory ideas in the NTIA response. I ended up doing this halfway through, and I would’ve done this sooner.
If you ever decide to write a doc properly explaining the situation with AI Safety to policymakers who read it, Scott Alexander’s Superintelligence FAQ is considered in high esteem, you could probably read it, think about how/why it was good at giving laymen a fair chance to understand the situation, and write a much shorter 1-pager yourself that’s optimized for the particular audience. I convinced both of my ~60-year-old parents to take AI safety seriously by asking them to read the AI chapter in Toby Ord’s The Precipice, so you might consider that instead.
Thanks for all of this! Here’s a response to your point about committees.
I agree that the committee process is extremely important. It’s especially important if you’re trying to push forward specific legislation.
For people who aren’t familiar with committees or why they’re important, here’s a quick summary of my current understanding (there may be a few mistakes):
When a bill gets introduced in the House or the Senate, it gets sent to a committee. The decision is made by the Speaker of the House or the priding officer in the Senate. In practice, however, they often defer to a non-partisan “parliamentarian” who specializes in figuring out which committee would be most appropriate. My impression is that this process is actually pretty legitimate and non-partisan in most cases(?).
It takes some degree of skill to be able to predict which committee(s) a bill is most likely to be referred to. Some bills are obvious (like an agriculture bill will go to an agriculture committee). In my opinion, artificial intelligence bills are often harder to predict. There is obviously no “AI committee”, and AI stuff can be argued to affect multiple areas. With all that in mind, I think it’s not too hard to narrow things down to ~1-3 likely committees in the House and ~1-3 likely committees in the Senate.
The most influential person in the committee is the committee chair. The committee chair is the highest-ranking member from the majority party (so in the House, all the committee chairs are currently Republicans; in the Senate, all the committee chairs are currently Democrats).
A bill cannot be brought to the House floor or the Senate floor (cannot be properly debated or voted on) until it has gone through committee. The committee is responsible for finalizing the text of the bill and then voting on whether or not they want the bill to advance to the chamber (House or Senate).
The committee chair typically has a lot of influence over the committee. The committee chair determines which bills get discussed in committee, for how long, etc. Also, committee chairs usually have a lot of “soft power”– members of Congress want to be in good standing with committee chairs. This means that committee chairs often have the ability to prevent certain legislation from getting out of committee.
If you’re trying to get legislation passed, it’s ideal to have the committee chair think favorably of that piece of legislation.
It’s also important to have at least one person on the committee as someone who is willing to “champion” the bill. This means they view the bill as a priority & be willing to say “hey, committee, I really think we should be talking about bill X.” A lot of bills die in committee because they were simply never prioritized.
If the committee chair brings the bill to a vote, and the majority of committee members vote in favor of the bill moving to the chamber, the bill can be discussed in the full chamber. Party leadership (Speaker of the House, Senate Majority Leader, etc.) typically play the most influential role in deciding which bills get discussed or voted on in the chambers.
Sometimes, bills get referred to multiple committees. This generally seems like “bad news” from the perspective of getting the bill passed, because it means that the bill has to get out of multiple committees. (Any single committee could essentially prevent the bill from being discussed in the chamber).
(If any readers are familiar with the committee process, please feel free to add more info or correct me if I’ve said anything inaccurate.)
> I think I would’ve written up a doc that explained my reasoning, documented the people I consulted with, documented the upside and downside risks I was aware of, and sent it out to some EAs.
Before I start criticizing, I would make it clear that I’m grateful for your work and I could not do better myself; I certainly did try, in fact I was one of the first in DC in 2018, but I could not do well since I was one of the many “socially-inept” people which are in fact a serious problem in DC (for the record: if you want to do AI policy, do not move to DC, first visit and have the people there judge your personality/charisma, the standards might be far higher or far lower than you might expect, they are now much better at testing people for fit than when I started 6 years ago).
I’m also grateful to see you put your work out there for review on Lesswrong, rather than staying quiet. I think the decision to attempt to be open vs. closed about AI policy work is vastly more complicated than most people in AI policy believe.
Your post is fantastic, especially the reflections.
You never mentioned the words “committee” or “chair” in this post. ?????????? Everything in Congress, other than elections and constituent calls, revolves around the congressional committees and particularly their chairmen. Is your model that congressional committees, just, aren’t important at all relative to party leadership in each chamber? If the balance of power has shifted that far by now, I wouldn’t know. Either way, congress is very much the kind of place where 20% of the congressmembers have >80% of the power, and the ones in the bottom 50% are the easiest to talk to, and their staffers exist to look important and talk to as many people as possible per day and make them feel heard, and their offices are focused on maintaining the appearance of being capable of substantially influencing legislation, in order to mitigate the risk that their voters and their professional network find out that they are in the bottom 50%. Over the centuries, Congress has become incredibly sophisticated at constructing mazes and leading people around. The Committee system is the first step to cutting through that and getting to where the bills are actually getting negotiated and written (primarily by lobbyists with de-facto personal ties to the office of the Chairs of the relevant committees and maybe the deputy chairs). Unless I’m wrong about this e.g. maybe a way larger share of policymaking power has accrued to the party leadership, who are even harder to meet with, or maybe the lobbyists from the big 5 tech companies are the main hotspot for tech-related policymaking in general including AI, and they meet with whoever they want, making the committee structure not very relevant to AI policy. It would have been great to hear more about the people you met at think tanks and the executive branch.
When it comes to foreign policy, which might be pretty important, a helpful way to look at it is like congress and other parliamentary bodies acting as a wall between domestic elites and the foreign policymaking institutions, and intelligence agencies as the main holders of real power. Obviously, these are people, and backdoor deals and revolving door employment is everywhere, so even this wall is fuzzy. But it is much more robust than, say, domestic policy (e.g. farm bills) where congress basically acts the conduit between elites and policy (e.g. like how most of the actual lawmaking work on capitol hill is done by lobbyists, not staffers). Intelligence agencies can easily bribe or infiltrate parliaments, and parliaments cannot easily bribe or infiltrate intelligence agencies. Authoritarian countries like China, on the other hand, don’t have real parliaments, and the strongman leader must mitigate the creep of rich domestic elites seeking policymaking influence (in reality it’s much more complex e.g. hybrid regimes, redirecting domestic elites to focus on local/provincial governments instead of the central/national government, etc your book might help with this but it’s important to note that books about intelligence agencies are products that need to optimize for entertainment in order to sell copies; books must be recommended by personal connections, and even then you never know, I might read and trust a book recommended to me by someone like Jason Matheny).
They want you to propose solutions, they get annoyed when people come to them with a new issue they know nothing about and expect them to be the one to think of solutions. They want you to do the work writing up the final product and then hand it to them. If they have any issue with it, they’ll rewrite parts of it or throw it in the recycle bin.
I’ve heard this characterized as “goldfish memory”. It’s important to note that many of the other 100 things on their priority list also have people trying to “expose and re-expose” them to ideas, and many staffers are hired for skill at pretending that they’re listening. I think you were correct to evaluate your work building relationships as more useful than this.
I disagree that the Overton window in DC, or even Congress, is as wide as your impression. This is both for the reasons stated above, and because it seems very likely (>95%) that military-adjacent people in both the US and China are actively pursuing AI for things like economic growth/stabilization, military applications like EW and nuclear-armed cruise missiles, or for the data processing required for modern information warfare. I agree that we seem to be in a period of unusually high open-mindedness and curiosity.
I think that DC is a very Moloch-infested place, resulting in an intense and pervasive culture of nihilism- a near-universal belief that Moloch is inevitable. Prolonged exposure to that environment (several years), where everyone around you thinks this way, and will permanently mark you as low-social-status if you ever reveal you are one of those people with hope for the world, likely (>90%) has intense psychological effects on the AI Safety people in DC.
Likewise, the best people will know the risks associated with having important conversations near smartphones in a world where people use AI for data science, but they don’t know you well enough to know whether you yourself will proceed to have important conversations about them near smartphones. They can’t heart-to-heart with you about the problem, because that would turn that conversation into an important one, and it would be near a smartphone.
internally screaming
If you ever decide to write a doc properly explaining the situation with AI Safety to policymakers who read it, Scott Alexander’s Superintelligence FAQ is considered in high esteem, you could probably read it, think about how/why it was good at giving laymen a fair chance to understand the situation, and write a much shorter 1-pager yourself that’s optimized for the particular audience. I convinced both of my ~60-year-old parents to take AI safety seriously by asking them to read the AI chapter in Toby Ord’s The Precipice, so you might consider that instead.
Thanks for all of this! Here’s a response to your point about committees.
I agree that the committee process is extremely important. It’s especially important if you’re trying to push forward specific legislation.
For people who aren’t familiar with committees or why they’re important, here’s a quick summary of my current understanding (there may be a few mistakes):
When a bill gets introduced in the House or the Senate, it gets sent to a committee. The decision is made by the Speaker of the House or the priding officer in the Senate. In practice, however, they often defer to a non-partisan “parliamentarian” who specializes in figuring out which committee would be most appropriate. My impression is that this process is actually pretty legitimate and non-partisan in most cases(?).
It takes some degree of skill to be able to predict which committee(s) a bill is most likely to be referred to. Some bills are obvious (like an agriculture bill will go to an agriculture committee). In my opinion, artificial intelligence bills are often harder to predict. There is obviously no “AI committee”, and AI stuff can be argued to affect multiple areas. With all that in mind, I think it’s not too hard to narrow things down to ~1-3 likely committees in the House and ~1-3 likely committees in the Senate.
The most influential person in the committee is the committee chair. The committee chair is the highest-ranking member from the majority party (so in the House, all the committee chairs are currently Republicans; in the Senate, all the committee chairs are currently Democrats).
A bill cannot be brought to the House floor or the Senate floor (cannot be properly debated or voted on) until it has gone through committee. The committee is responsible for finalizing the text of the bill and then voting on whether or not they want the bill to advance to the chamber (House or Senate).
The committee chair typically has a lot of influence over the committee. The committee chair determines which bills get discussed in committee, for how long, etc. Also, committee chairs usually have a lot of “soft power”– members of Congress want to be in good standing with committee chairs. This means that committee chairs often have the ability to prevent certain legislation from getting out of committee.
If you’re trying to get legislation passed, it’s ideal to have the committee chair think favorably of that piece of legislation.
It’s also important to have at least one person on the committee as someone who is willing to “champion” the bill. This means they view the bill as a priority & be willing to say “hey, committee, I really think we should be talking about bill X.” A lot of bills die in committee because they were simply never prioritized.
If the committee chair brings the bill to a vote, and the majority of committee members vote in favor of the bill moving to the chamber, the bill can be discussed in the full chamber. Party leadership (Speaker of the House, Senate Majority Leader, etc.) typically play the most influential role in deciding which bills get discussed or voted on in the chambers.
Sometimes, bills get referred to multiple committees. This generally seems like “bad news” from the perspective of getting the bill passed, because it means that the bill has to get out of multiple committees. (Any single committee could essentially prevent the bill from being discussed in the chamber).
(If any readers are familiar with the committee process, please feel free to add more info or correct me if I’ve said anything inaccurate.)
Can you please explain what this means?