Mass_Driver

Karma: 3,244

Mass_Driver May 30, 2025, 9:04 PM
2 points
0
in reply to: Charlie Steiner’s comment on: Orphaned Policies (Post 5 of 6 on AI Governance)
Transparency is less neglected than some other topics—check out HR 5539 (Transparent by Design Act), S 3312 (AIRIA), and HR 6881 (AI Foundation Model Transparency Act).
There’s room for a little bit more useful drafting work here, but I wouldn’t call it orphaned, exactly.

Mass_Driver May 29, 2025, 3:02 PM
3 points
2
in reply to: ErickBall’s comment on: Shift Resources to Advocacy Now (Post 4 of 6 on AI Governance)
I could carry on debating the pros and cons of the EO with you, but I think my real point is that bipartisan advocacy is harmless. You shouldn’t worry that bipartisan advocacy will backfire, so we can’t justify engaging in no advocacy at all out of fear that advocacy might backfire.
If you believe strongly enough in the merits of working with one party to be confident that it won’t backfire, fine, I won’t stop you—but we should all be able to agree that more bipartisan advocacy would be good, even if we disagree about how valuable one-party advocacy is.

Mass_Driver May 28, 2025, 7:39 PM
7 points
2
in reply to: ErickBall’s comment on: Shift Resources to Advocacy Now (Post 4 of 6 on AI Governance)
Yes, CAIP’s strategy was primarily to get the legislation ready, talk to people about it, and prepare for a crisis. We also encouraged people to pass our legislation immediately, but I was not especially optimistic about the odds that they would agree to do so.
I don’t object to people pushing legislators from both parties to act more quickly...but you have to honor their decision if they say “no,” no matter how frustrating that is or how worried you are about the near-term future, because trying to do an end-run around their authority will quickly and predictably backfire.
In my opinion, going behind legislators’ backs to the Biden administration was particularly unhelpful for the Biden AI EO, because the contents of that EO would have led to only a small reduction in catastrophic risk—it would be nice to require reports on the results of red-teaming, but the EO by itself wouldn’t have stopped companies from reporting that their models seemed risky and then releasing them anyway. We would have needed to follow up on the EO and enact additional policies in order to have a reasonable chance of survival, but proceeding via unilateral executive action had some tendency to undermine our ability to get those additional policies passed, so it’s not clear to me what the overall theory of change was for rushing forward with an EO.

Mass_Driver May 28, 2025, 6:52 PM
2 points
1
in reply to: ErickBall’s comment on: Shift Resources to Advocacy Now (Post 4 of 6 on AI Governance)
I think it’s worth distinguishing between “this AI policy could be slightly inconvenient for America’s overall geopolitical strategy” and “this AI policy is so bad for America’s AI arms race that we’re going to lose a shooting war with China.”
The former is a political problem that advocates need to find a strategy to cope with, but it’s not a reason not to do advocacy—we should be willing to trade away a little bit of American influence in order to avoid a large risk of civilizational collapse from misaligned AI.
If you’re earnestly worried about maximizing American influence, there are much better strategies for making that happen than trying to make sure that we have zero AI regulations. You could repeal the Jones Act, you could have a stable tariff regime, you could fix the visa system, you could fund either the CHIPS Act or a replacement for the CHIPS Act, you could give BIS more funding to go after chip smugglers, and so on.
I think the “concern” about the harmful geopolitical effects of moderate AI regulation is mostly opportunistic political theater by companies who would prefer to remain unregulated—there’s a notable absence of serious international relations scholars or national security experts who are coming out in favor of zero AI safety regulation as a geopolitical tool. At most, some experts might be pushing for easier land use and environmental approvals, which are not in conflict with the regulations that organizations like CAIP are pushing for.

Mass_Driver May 28, 2025, 6:31 PM
6 points
−1
in reply to: ErickBall’s comment on: Shift Resources to Advocacy Now (Post 4 of 6 on AI Governance)
I tend to agree with John—I think the Republican opposition is mostly a mixture of general hostility toward regulation as a tool and a desire to publicly reject the Biden Administration and all its works. My experience in talking with Republican staffers, even in 2025, is that they weren’t hostile to the idea of reducing catastrophic risks from AI; they’re just waiting to see what the Trump Administration’s official position is on the issue. We’ll know more about that when we see the OSTP AI Action Plan in July.
Part of the problem with AI safety advocacy in 2022 − 2024 was that it ignored my advice to “insist that the bills you endorse have co-sponsors from both parties.” By definition, an executive order is not bipartisan. You can frame the Biden EO as general technocracy if you like, but it’s still careless to push for an EO that’s only supported by one party. If you want to avoid alienating Republicans (and you should want to avoid doing that), then you need to make sure you have at least some Republicans willing to go on record as publicly endorsing your policy proposals before you enact them.

Mass_Driver May 28, 2025, 12:37 PM
4 points
0
in reply to: cdt’s comment on: Shift Resources to Advocacy Now (Post 4 of 6 on AI Governance)
Hmm. There are lots of valuable advocacy targets besides literal regulation; advocates might try to pass laws, win court cases, sustain a boycott, amend a corporate charter, amend a voluntary industry standard at NIST, and so on. I’ll discuss several examples in post #5.
I suppose advocacy might also have some side benefits like teaching us more about politics, helping us bond as a community, or giving us a sense of efficacy, but it’s not obvious to me that any of those are of comparable importance to reducing the likelihood that AI developers release misaligned superintelligence.
Does that answer your question? If not, please let me know more about what you mean and I’ll try to elaborate.

Mass_Driver May 28, 2025, 7:56 AM
7 points
0
in reply to: RogerDearnaley’s comment on: Shift Resources to Advocacy Now (Post 4 of 6 on AI Governance)
I have no complaints at all about technical AI research! Indeed, I agree with you that industry will spontaneously adopt most of the good technical AI safety ideas shortly after they’re invented.
I’m arguing about the relative merits of AI governance research vs. AI governance advocacy. This probably wasn’t clear if you’re just jumping in at this point of the sequence, for which I apologize.

Mass_Driver May 26, 2025, 1:58 PM
4 points
2
in reply to: Charbel-Raphaël’s comment on: We’re Not Advertising Enough (Post 3 of 6 on AI Governance)
Thanks! If you are directly and literally comparing the utility of two or three or four concrete policy options, that’s more valuable than general academic research—it’s at least level 3, and possibly better if you write up your conclusions in a format that can be easily digested by policy wonks (level 2) or by politicians (level 1).
Unfortunately, most academic research weighs the pros and cons of general categories of policies and then offers no concrete recommendations, so a lot of what looks like “comparing different options” at first glance is not immediately valuable as a tool for picking the best currently available option.
Brainstorming new policy options to further explore the available space is at least a level 3 activity, and possibly level 2 (if you write up some of those options in enough detail that someone could advocate for them). However, as I’ll argue in the fifth post in this sequence, we already have a dozen “orphaned” policy proposals that nobody is drafting up or advocating for, so unless and until this changes, we’ll get limited value from having an even larger stack of orphaned policies. Even if you find a policy that’s an order of magnitude better than previous ideas, that policy still doesn’t generate any utility until someone goes and does the political work needed to turn it into law.

Mass_Driver May 26, 2025, 1:53 PM
15 points
11
in reply to: Josh You’s comment on: We’re Not Advertising Enough (Post 3 of 6 on AI Governance)
I agree that people like Bengio can be very valuable assets for AI safety advocacy, although there are diminishing marginal returns—the first computer supergenius who likes your policies is transformative; the third is helpful; the tenth is mostly just a statistic in a survey and will not meaningfully change, e.g., journalists or staffers’ opinions about an issue.
If you think that technology or history will move far enough ahead that people like Bengio and Hinton will lose their relevance, then it might be a good idea to try to convince the next Bengio to support some AI safety policies. If that’s your strategy, then you should develop a short list of people who might be the next Bengio, and then go find them and talk to them in person. Once you’ve identified some leading young computer scientists and some questions that they’re uncertain and curious about, then you can go do research to try to help convince them to take a particular stance on those questions.
Just publishing AI governance research of general academic interest is very far removed from the goal of recruiting computer science superstars for AI x-risk advocacy.

Mass_Driver May 26, 2025, 1:49 PM
13 points
7
in reply to: Josh You’s comment on: We’re Not Advertising Enough (Post 3 of 6 on AI Governance)
I would call the main AI 2027 report an advocacy document—its function is to tell a compelling story about how AI could cause large harms in the very near future. Although the authors claim that a “slowdown” is not their real policy recommendation, the entire structure of the piece (a good ending with a green button and a bad ending with a red button) strongly pushes readers in the direction of thinking that it would be good if the White House “centralizes compute and brings in external oversight.” The story contains enough concrete details that particular White House officials (like the VP) could imagine doing this, so whether the authors intended it or not, the document acts like an advocacy pitch that asks those officials to take these actions.
The research supporting the AI 2027 report is pushed back to a secondary page that readers can check in on if they’re still interested after they finish reading the advocacy document. This is good practice, and I would like to see more researchers adopt this type of structure. Unfortunately, most of our research today doesn’t include an advocacy document at all, let alone as the primary focus of media engagement.

Mass_Driver May 26, 2025, 1:31 PM
5 points
9
in reply to: Knight Lee’s comment on: We’re Not Advertising Enough (Post 3 of 6 on AI Governance)
There’s actually very little shooing! It can be a challenge to get on the calendar of the highest-ranking staffers, because they’re very busy, but once you’re in the room they’re almost always happy to chat. AI is a sexy topic, even in DC, and if you’re coming in with a public safety concern then that’s something interesting that they haven’t heard 100 times already.
I’ll talk more about this in the next post, but there’s a difference between “researching to identify a miracle strategy for more efficient advocacy” and “researching because it’s of solid academic interest.” The first option mostly involves learning a lot about politics! If you’re not intensely studying the preferences of particular Senators, their donors, their chiefs of staff, etc., then how can you hope to identify a strategy that will be an order of magnitude more politically effective than other strategies?
So, I would never criticize anyone for saying “Gosh, it doesn’t seem like existing strategies can succeed in the time available; let me look for a much better strategy.” However, that doesn’t seem to be the main purpose or effect of most AI governance research. They’re not explicitly considering the effectiveness of different strategies or proposing new political strategies; they’re just discussing the risks and benefits of various policies in a general way—and it’s not the quality of our policies that’s the bottleneck.

Mass_Driver May 23, 2025, 7:04 PM
2 points
0
in reply to: PeterMcCluskey’s comment on: Please Donate to CAIP (Post 1 of 3 on AI Governance)
Thank you!! Very much appreciated.

Mass_Driver May 8, 2025, 9:19 PM
3 points
0
in reply to: MichaelDickens’s comment on: Please Donate to CAIP (Post 1 of 3 on AI Governance)
Thank you!!

Mass_Driver May 8, 2025, 12:57 AM
6 points
1
in reply to: MalcolmMcLeod’s comment on: Please Donate to CAIP (Post 1 of 3 on AI Governance)
Thanks, Malcolm, I appreciate it.
You can donate to Manifund if you like, which provides a sort of compact in that if we raise less than $50k, we don’t get any of it. If there are several other people who feel the same way as you do, then we can find a way to run a similar campaign that has a higher ‘minimum.’

Mass_Driver May 8, 2025, 12:07 AM
2 points
0
in reply to: Austin Chen’s comment on: Please Donate to CAIP (Post 1 of 3 on AI Governance)
Thanks to both of you; I appreciate the support! We think that about 40% of our work is (c)(3) compatible, and we’re in the process of spinning up a companion (c)(3) org to take over responsibility for funding that work, but we don’t know when the IRS will finish approving our application.
In the meantime, if you have any hesitation about donating to us directly, then please use Manifund. Otherwise, direct donations are slightly more useful to us (and, apparently, easier for Manifund).

Mass_Driver May 7, 2025, 6:52 PM
19 points
3
in reply to: TsviBT’s comment on: Please Donate to CAIP (Post 1 of 3 on AI Governance)
Fair question—although it would be speculation on my part, since we haven’t been actively studying attitudes toward a moratorium. You might do better to ask Pause AI. That said, I would think you’d need something on the scale of the Vietnam War protests to get a blanket unilateral moratorium on advanced AI inside the Overton window—protests large enough to generate major headlines on most days that continue for months on end.
If you can get very credible evidence that the UK, the EU, and China would all agree to join such a moratorium, then a smaller protest movement might be good enough, since one of the major reasons to oppose a moratorium is that it would allow rivals to pull ahead of the US. Industry testimony would also be helpful, but only if it’s sincere; there was a fair amount of skepticism when big tech companies came to DC to talk about the need for AI regulations, and that skepticism was probably warranted, since most of those companies have now reversed their positions, with even Anthropic mostly opposing CA SB 1047.
You can get a sense of how much other groups are spending on lobbying and campaign contributions from this article: https://www.nbcnews.com/investigations/lobbyists-spent-record-42-billion-2023-federal-lawmakers-rcna135943. The figures listed are probably an underestimate of the actual cost by a factor of at least 3x to 10x, because in addition to actual lobbyists, you also usually pay for some think tanks, some support staff, media campaigns, and so on; the publicly disclosed figures that get used in these articles are based only on the salaries of the lobbyists. There may also be soft money donations or other hidden contributions that I’m not aware of. Lobbying is also most effective when you do it for multiple years in a row so that you can build relationships and people know who you are—it’s not like you can just buy influence overnight for a fixed amount of money; you need to be spending that money in a way that feels appropriate to DC insiders.

Mass_Driver Jun 2, 2024, 10:26 PM
2 points
0
on: Arguments about fast takeoff
I just came here to point out that even nuclear weapons were a slow takeoff in terms of their impact on geopolitics and specific wars. American nuclear attacks on Hiroshima and Nagasaki were useful but not necessarily decisive in ending the war on Japan; some historians argue that the Russian invasion of Japanese-occupied Manchuria, the firebombing of Japanese cities with massive conventional bombers, and the ongoing starvation of the Japanese population due to an increasingly successful blockade were at least as influential in the Japanese decision to surrender.
After 1945, the American public had no stomach for nuclear attacks on enough ‘enemy’ civilians to actually cause large countries like the USSR or China to surrender, and nuclear weapons were too expensive and too rare to use them to wipe out large enemy armies—the 300 nukes America had stockpiled at the start of the Korean War in 1950 would not necessarily have been enough to kill the 3 million dispersed Chinese soldiers who actually fought in Korea, let alone the millions more who would likely have volunteered to retaliate against a nuclear attack.
The Soviet Union had a similarly-sized nuclear stockpile and no way to deliver it to the United States or even to the territory of key US allies; the only practical delivery vehicle at that time was via heavy bomber, and the Soviet Union had no heavy bomber force that could realistically penetrate western air defense systems—hence the Nike anti-aircraft missiles rusting along ridgelines near the California coast and the early warning stations dotting the Canadian wilderness. If you can shoot their bombers down before they can reach your cities, then they can’t actually win a nuclear war against you.
Nukes didn’t become a complete gamechanger until the late 1950s, when the increased yields from hydrogen bombs and the increased range from ICBMs created a truly credible threat of annihilation.

Mass_Driver Jun 12, 2022, 7:14 AM
2 points
0
in reply to: TurnTrout’s comment on: AGI Ruin: A List of Lethalities
I suspect we’re talking about two different things.
If you just naively program a super-intelligent AI to satisfice a goal, then, sure, most of the candidate pathways to satisfice will involve accruing a lot of some type of power, because power is useful for achieving goals. That’s a valid point, and it’s important to understand that merely switching from optimizers to satisficers won’t adequately protect us against overly ambitious AIs.
However, that doesn’t mean that it’s futile to explicitly penalize most (but not literally all) of the paths that the AI could take to accumulate too much power. Suppose you adequately penalize all of the tactics that would have catastrophic side effects except for, oh, I don’t know, cornering the world market on all types of grain, because you forgot to think of that one particular path to power. Would most of the candidate plans that the AI submits for human approval then turn out to involve secretly cornering the grain market?
I don’t see why they would. All else being equal, sure, the most powerful tactic available is going to be somewhat more attractive than other, less powerful tactics. But how much more attractive? Would an AI generate 1,000x more plans that involve one specific globally disruptive tactic like cornering the grain market vs. all of the various tactics that are only locally disruptive, like publishing propaganda or hiring away competing researchers or hacking into competing research labs or interrupting the flow of electricity to those labs? 10x more such plans? 2x more such plans? I don’t think that’s the kind of estimate you can make just by gesturing at basic principles of game theory; you’d need some concrete domain knowledge about the AI’s specific planning algorithms.
If the truly dangerous plans are only 10x more common on the initial brainstorming list, then we can probably make sure those plans aren’t chosen by filtering for plans that the AI rates as safe and transparent. We can use 3 different AIs on airgapped equipment—one that only “cares” about evaluating whether a plan is safe (i.e., would the plan cause unacceptably large changes if implemented?), one that only “cares” about evaluating whether a plan is transparent (i.e., can the plan be explained to humans so that the humans will have an accurate understanding of what actions will be taken as part of the plan?), and one that generates plans that satisfice for a substantive goal (in this case, shutting down opposing AI research).
Of the plans that are approved by all 3 AIs, most of them probably won’t involve accumulating excessive amounts of power, because usually if you accumulate too much power, then that is (a) unsafe, and (b) likely to involve lying to ‘your’ human programmers.
If powerful plans are 10x more common on the initial brainstorming list, then you need filters for safety and/or transparency that are at least 90% effective at catching unacceptably powerful plans in order to get back up to a 50% chance of humanity surviving. If powerful plans are more than 10x more common, then you need a set of filters that’s more reliable than 90%. I don’t think we have any idea what those numbers are yet, but I do think it’s worth trying to reduce how common it is for excessively powerful plans to show up on the initial brainstorming list, and I think we can do that by training AIs to recognize dangerously disruptive plans and to try to avoid those types of plans. It’s better to at least try to get AIs to engage with the concept of “this plan is too disruptive” then to throw up our hands and say, “Oh, power is an attractor in game theory space, so there’s no possible way to get brilliant AIs that don’t seize infinite power.”

Mass_Driver Jun 7, 2022, 6:46 AM
7 points
4
in reply to: Rob Bensinger’s comment on: AGI Ruin: A List of Lethalities
Sure, the metaphor is strained because natural selection doesn’t have feelings, so it’s never going to feel satisfied, because it’s never going to feel anything. For whatever it’s worth, I didn’t pick that metaphor; Eliezer mentions contraception in his original post.
As I understand it, the point of bringing up contraception is to show that when you move from one level of intelligence to another, much higher level of intelligence, then the more intelligent agent can wind up optimizing for values that would be anathema to the less intelligent agents, even if the less intelligent agents have done everything they can to pass along their values. My objection to this illustration is that I don’t think anyone’s demonstrated that human goals could plausibly be described as “anathema” to natural selection. Overall, humans are pursuing a set of goals that are relatively well-aligned with natural selection’s pseudo-goals.

Mass_Driver Jun 7, 2022, 6:37 AM
5 points
0
AF
in reply to: Chris_Leong’s comment on: AGI Ruin: A List of Lethalities
One of my assumptions is that it’s possible to design a “satisficing” engine—an algorithm that generates candidate proposals for a fixed number of cycles, and then, assuming at least one proposal with estimated utility greater than X has been generated within that amount of time, selects one of the qualifying proposals at random. If there are no qualifying candidates, the AI takes no action.
If you have a straightforward optimizer that always returns the action with the highest expected utility, then, yeah, you only have to miss one “cheat” that improves “official” utility at the expense of murdering everyone everywhere and then we all die. But if you have a satisficer, then as long as some of the qualifying plans don’t kill everyone, there’s a reasonable chance that the AI will pick one of those plans. Even if you forget to explicitly penalize one of the pathways to disaster, there’s no special reason why that one pathway would show up in a large majority of the AI’s candidate plans.