Mass_Driver

Karma: 3,646

Mass_Driver 3 Sep 2026 23:30 UTC
8 points
7
in reply to: denkenberger’s comment on: PauseAI Has ‘officially disendorsed’ PauseAI-US
If you actually donate most of your earnings, and you actually are pretty replaceable as a capabilities researcher, and you actually find effective charities, then you’re an exception to the rule and I don’t think you’re a bad person. However, it’s one thing to speculate that a basis point of x-risk might be worth about $300M; it’s another to come up with trustworthy evidence that a particular charity is worth at least one basis point per $300M. If you’re going to take a day job that involves destroying the world, you’d better be damn sure that your ethical offsets actually function the way you intended.

Mass_Driver 3 Sep 2026 2:05 UTC
31 points
21
on: PauseAI Has ‘officially disendorsed’ PauseAI-US
Speaking entirely in my personal capacity, I noticed that direct quotes from Holly Elmore are mostly missing from this piece, so I thought I’d add one that resonated with me:
“It’s wrong to work for the AI companies and it’s wrong to defend them. It’s wrong to have friends who have a stake in disempowering or destroying humanity. This is not something we can be nice about and discuss over tea. There is no ‘right’ way to deliver this vital critique— the reason they don’t want to hear it is that they are doing wrong.”
https://x.com/ilex_ulmus/status/2070746368910520506
I don’t draw my moral lines in exactly the same places that Holly does—I think it can be reasonable to work for an AI company if you’re working almost exclusively on safety or if you have a detailed, concrete, public commitment about how you will use your leverage for safety. I genuinely value much of the safety work being done by frontier developers. I also think it’s fine to attend conferences and other professional events with AI capabilities researchers and to negotiate with them—half the point of professional events is to provide an opportunity for dialogue among people who are not personal friends.
That said, I do think there are moral lines. I believe that working on frontier AI capabilities is usually wrong, just like it it would usually be wrong to work on mining dirty coal or locking people up without due process or bundling subprime mortgages or marketing tobacco to kids. If you go out and get a job like that and your children aren’t starving, then in my opinion, that’s pretty strong evidence that you have bad character. It’s always hard to see into other people’s hearts, and I think we should have a very high bar for accusing others of having bad character, but for me, working on frontier AI capabilities in 2026 typically clears that bar.
I can’t and won’t defend the details of Holly’s communication style—even she would admit that she’s more abrasive than necessary—but I think her core point is more true than false: it’s almost always wrong to work on AI capabilities, and people who choose to do so should usually feel ashamed. I don’t see it as useful to try to erase this core point from the AI safety movement’s public messaging. Instead, I think that both conflict theory and mistake theory should be part of our toolkit for addressing the problem of unsafe AI.

Mass_Driver 13 Jul 2026 5:31 UTC
29 points
8
on: The current bottleneck is political will, not research
Thanks, Charbel, excellent analysis! Speaking in my personal capacity, I agree with everything you’re saying here and believe it’s extremely important. The only feature I’d like to try to improve is the specific levels of your ‘policymaker awareness pipeline.’ I think your definitions of these levels jump around somewhat jaggedly and would be more useful and easier to measure if they were a little smoother. Here’s my attempt:
- Level 0 (Clueless): Not aware of AI x-risks
- Level 1 (Uninterested): Aware of the concept of AI x-risks, but not aware that serious people think they’re a big problem.
- Level 2 (Interested): Aware that serious people think AI x-risks are a big problem
- Level 3 (Privately Concerned): Privately agree that AI x-risks are a big problem.
- Level 4 (Publicly Concerned): On the public record as saying that AI x-risks are a big problem.
- Level 5 (Active): Has taken at least one step toward reducing AI x-risks
- Level 6 (Committed): Regularly pushes to reduce AI x-risks, even when this costs significant effort or political capital
- Level 7 (Champion): Inspires, persuades, or directs other politicians to improve their stances on AI x-risks

Mass_Driver 8 Apr 2026 6:55 UTC
5 points
−1
on: The hard part isn’t noticing when papers are bad, it’s deciding what to do afterwards
I agree that steering toward truth is better than dunking on opponents, and I think your first and third suggestions for how to encourage steering toward truth are quite reasonable.
I’m not convinced that, as a rule of thumb, it makes sense to gloss over formatting errors or missing citations. Of course there are examples of critiques about formatting or citations that are thoughtless and unhelpful dunks, but it’s not obvious to me that most such critiques are unhelpful.
In particular, if the concept of “formatting” is broad enough to include things like the choice of title, choice of section headers, relative order and hierarchy of sections, etc., then I often see papers that are so badly formatted that it’s not clear what if anything the author is trying to say. Similarly, a poorly formatted graph or chart might fail to convey its key points or make digesting these points so difficult as to not be worth the effort for a typical reader.
With regard to citations, it’s one thing to complain that a paper is only citing two out of three of the relevant pieces of prior work—but it’s another thing to complain that a paper seems blissfully unaware of an entire relevant body of prior work. This is especially problematic if the prior work persuasively establishes some limitations on or reasons to be skeptical of the author’s preferred data or methodology.
I’m curious to what extent you agree with these counterpoints (in which case we’re haggling over semantics) and to what extent you think that reviewers really should refrain from complaining about missing structure and missing acknowledgements/caveats (in which case I’d love to hear more about why.)

Mass_Driver 8 Jan 2026 5:56 UTC
13 points
7
in reply to: davekasten’s comment on: davekasten’s Shortform
I hope you’re right that job losses in more ‘stable’ fields will catalyze interest in a constructive response, but I was surprised over the last 20 years or so as the market power of workers in various traditionally stable industries collapsed for mundane economic reasons and not much changed in the policy world. Professors, lawyers, accountants, civil servants, and even some types of physicians have all been squeezed fairly heavily in the US just from globalization, monopolization, and deregulation. There was some brief pushback around the time of Occupy Wall Street, and then after that increased job insecurity became part of the new status quo.
Entry level software engineers are now facing serious pressure. It’s debatable whether this is from pandemic-era over-hiring or from AI, but until last year “software engineering” was the paradigmatic example of a job more stable than art, to the point where artists went to coding bootcamp if they wanted to sell out. Now bootcamps seem mostly dead, but I don’t hear serious cries to save them.

Mass_Driver 13 Nov 2025 0:18 UTC
3 points
0
in reply to: avturchin’s comment on: Warning Aliens About the Dangerous AI We Might Create
Yeah, but have you done a back of the envelope calculation here, or has anyone else? What size target could we hit in the Andromeda galaxy using, e.g., $50 million at our current tech levels, and how long could we transmit for? How large of a receiver would that target need to have pointing toward us in order to receive the message with anything like reasonable fidelity? If our message is focused no more tightly than on “a star,” then would the receivers need an antenna the size of a solar system to pick it up? If not, why not?

I’m not sure codebreaking is a reasonable test of a supposedly universal language. A coded message has some content that you know would make sense if the code can be broken. By contrast, a CosmicOS message might or might not have any content that anyone else would be able to absorb. Consider the difference between, e.g., a Chinese transmission sent in the clear, and an English transmission sent using sophisticated encryption. If you’re an English speaker who’s never been exposed to even the concept of a logographic writing system, then it’s not obvious to me that it will be easier to make sense of the plaintext Chinese message than the encrypted English message. I think we should test that hypothesis before we invest in an enormous transmitter.

I’m not sure what your comment “if we will start discussing it, we will not reach consensus for many years” implies about your interest in this conversation. If you don’t see a discussion on this topic as valuable, that’s fine, and I won’t take up any more of your time.

Mass_Driver 12 Nov 2025 17:55 UTC
9 points
4
on: Warning Aliens About the Dangerous AI We Might Create
I think this is a promising strategy that deserves more investigation. Your game theory analysis of dark forest-type situations is particularly compelling; thank you for sharing it. I have two main questions: (1) to what extent is this technically feasible, and (2) how politically costly would the weirdness of the proposal be?

For technical feasibility, I was very surprised to hear you suggest targeting the Andromeda Galaxy. I agree that in principle the nearest stars are more likely to already have whatever data they might want about Earth, but I think of “the nearest stars” as being within 50 light-years or so, not as including the entire Milky Way. Can you explain why you think we’d be able to send any message at all to the Andromeda Galaxy in the next few years, or why an alien civilization 1,000 light-years away in a different part of the Milky Way would most likely be able to passively gather enough data on Earth to draw their own conclusions about us without the need for a warning?

The other part of the technical feasibility question is whether constructed languages like CosmicOS actually work. Has anyone done testing to see whether, e.g., physicists with no prior exposure to the language and no reference guides are able to successfully decipher messages in CosmicOS?

Politically, I’d like to see focus groups and polling on the proposal. Does the general American public approve or disapprove of such warnings? Do they think it’s important or unimportant? What about astronomers, or Congressional staffers, or NASA employees? Yes, this is a weird idea, but the details could turn out to matter in terms of whether it’s so weird that there’s a high risk of burning significant amounts of credibility for the AI safety movement as a whole.

Mass_Driver 9 Oct 2025 17:18 UTC
2 points
0
in reply to: RamblinDash’s comment on: Four Questions to Refine Your Policy Proposal
I am literally a tort litigator in the United States! I worked for several years as a personal injury and product safety litigator.
Although the American legal system holds out “use reasonable care” as its official standard, in practice this gets further defined and specified based on rules found in OSHA, IEEE, or whatever the professional code of practice is for the relevant industry. As a plaintiff’s attorney, if you can’t point to a specific rule or norm that the defendant broke, you’re extremely unlikely to recover any damages. The violation of the rule or the norm is taken as very persuasive evidence that the defendant didn’t use reasonable care—and, conversely, if you can’t point to any crisp rule violation, that’s usually taken as very persuasive evidence that the defendant did use reasonable care.

Mass_Driver 10 Sep 2025 20:43 UTC
2 points
0
in reply to: eggsyntax’s comment on: MAGA speakers at NatCon were mostly against AI
Agreed; well said.

Mass_Driver 8 Sep 2025 19:14 UTC
66 points
70
in reply to: Seth Herd’s comment on: MAGA speakers at NatCon were mostly against AI
Yes, that’s exactly right, we do. That’s what it means to be an ally rather than a friend. America allied with the Soviet Union in World War 2; this is no different. When someone earnestly offers to help you literally save the world, you hold your nose and shake their hand.

Mass_Driver 20 Jul 2025 21:32 UTC
2 points
0
in reply to: PeterMcCluskey’s comment on: Mainstream Grantmaking Expertise (Post 7 of 7 on AI Governance)
Fair enough; we certainly paid much less than people could make in the private sector at CAIP, for essentially that reason. It’s good for nonprofit staff to have some skin in the game.
My suggestion to consider more competitive wages is mostly a response to Oliver suggesting that LTFF has had a serious and long-term challenge in hiring as many people as they would need to fully accomplish their mission.

Mass_Driver 9 Jul 2025 22:55 UTC
6 points
3
in reply to: habryka’s comment on: Mikhail Samin’s Shortform
Part of the distinction I try to draw in my sequence is that the median person at CSET or RAND is not “in politics” at all. They’re mostly researchers at think tanks, writing academic-style papers about what kinds of policies would be theoretically good for someone to adopt. Their work is somewhat more applied/concrete than the work of, e.g., a median political science professor at a state university, but not by a wide margin.
If you want political experts—and you should—you have to go talk to people who have worked on political campaigns, served in the government, or led advocacy organizations whose mission is to convince specific politicians to do specific things. This is not the same thing as a policy expert.
For what it’s worth, I do think OpenPhil and other large EA grantmakers should be hiring many more people. Hiring any one person too quickly is usually a mistake, but making sure that you have several job openings posted at any given time (each of which you vet carefully) is not.

Mass_Driver 9 Jul 2025 21:29 UTC
9 points
0
in reply to: habryka’s comment on: Mikhail Samin’s Shortform
I’m the author of the LW post being signal-boosted. I sincerely appreciate Oliver’s engagement with these critiques, and I also firmly disagree with his blanket dismissal of the value of “standard practices.”
As I argue in the 7th post in the linked sequence, I think OpenPhil and others are leaving serious value on the table by not adopting some of the standard grant evaluation practices used at other philanthropies, and I don’t think they can reasonably claim to have considered and rejected them—instead the evidence strongly suggests that they’re (a) mostly unaware of these practices due to not having brought in enough people with mainstream expertise, and (b) quickly deciding that anything that seems unfamiliar or uncomfortable “doesn’t make sense” and can therefore be safely ignored.
We have a lot of very smart people in the movement, as Oliver correctly points out, and general intelligence can get you pretty far in life, but Washington, DC is an intensely competitive environment that’s full of other very smart people. If you try to compete here with your wits alone while not understanding how politics works, you’re almost certainly going to lose.

Mass_Driver 1 Jul 2025 15:10 UTC
3 points
0
in reply to: habryka’s comment on: Mainstream Grantmaking Expertise (Post 7 of 7 on AI Governance)
These are important points, and I’m glad you’re bringing them up!
1. Is spending a lot of time to assess new grantmakers merely distressing (but still net positive in terms of extending your total grantmaking ability), or is it actually causing you to lose time in expectation? In other words, if you spend 40 hours recruiting and assessing candidates, does one of those candidates then go on to do 100+ hours of useful grantmaking work? Or is it more like 20 hours of useful grantmaking work?
2. How closely connected is the shortage of people willing to be full-time grantmakers with an expectation that grantmakers will already be fluent in technical AI safety when they start work? I could imagine that people who could otherwise be working for ARC or Anthropic would be very difficult to lure away full-time, but there’s an entire field of mainstream philanthropic foundations that mostly have full-time staff working on their grants. Could we hire some of those grantmakers full time to lend their general grantmaking expertise, evaluating things like budgets and org charts and performance targets, while relying on part-time advisors to provide technical expertise about the details of AI safety research? If not, why not?
3. What do you see as the most likely or most important negative consequence if grantmakers try to offer highly competitive salaries? Is this something that your funders have literally refused to pay for, or are you worried about being criticized for it (by whom? what consequences would follow from that criticism?), or does it just generally increase team members’ anxiety levels, or what exactly is the downside? I have definitely seen some of this paranoia you’re talking about, so it’s a real problem, but I wonder if it’s worth accepting the costs associated with paying highly competitive salaries in order to attract more and better people. It’s also worth noting that ‘highly competitive nonprofit salaries’ are still lower than ‘highly competitive tech salaries,’ probably by a factor of about 3. You can get top-notch grantmaking talent for much less than the price of top-notch computer engineering talent.

Mass_Driver 29 Jun 2025 6:58 UTC
7 points
0
in reply to: daijin’s comment on: I’m scared.
I mostly just got older and therefore calmer. I’ve crossed off most of the highest-priority items from my bucket list, so while I would prefer to continue living for a good long while, my personal death and/or defeat doesn’t seem so catastrophically bad anymore, and to cope with the loss of civilization/humanity I read a lot of history and sci-fi and anthropology and other works that help me zoom out and see that there has already been great loss and that while I do want to spend my resources fighting to reduce the risk of that loss, it’s not something I need to spend a lot of time or energy personally suffering over, especially not in advance. Worry is interest paid on trouble before it’s due.

Mass_Driver 29 Jun 2025 2:26 UTC
4 points
0
in reply to: Charbel-Raphaël’s comment on: Orphaned Policies (Post 5 of 6 on AI Governance)
Interesting thoughts; thanks for sharing, and for your work at CeSIA.
I’ve put some work into building coordination among US AI safety advocates, and it’s been somewhat helpful, but there are limits to how much we can expect discussions about coordination to lead to unified action because different organizations have different funders, different principles, and different interests. Merely sharing information about what different groups are working on will not spontaneously cause those groups to pick a single task and pivot to supporting it.

Mass_Driver 29 Jun 2025 2:21 UTC
2 points
0
in reply to: Charbel-Raphaël’s comment on: Political Funding Expertise (Post 6 of 7 on AI Governance)
I suppose I was speaking too loosely—thank you for flagging that!
I don’t mean that it’s literally impossible to assess whether AI governance grants have been successful—only that doing so requires somewhat more deliberate effort than it does for most other types of grants, and that there is relatively less in the way of established infrastructure to support such measurements in the field of AI governance.
If you run an anti-malaria program, there’s a consensus about at least the broad strokes of what you’re supposed to measure (i.e., malaria cases), and you’ll get at least some useful information about that metric just from running your program and honestly recording what your program officers observe as they deliver medication. If your bed nets are radically reducing the incidence of malaria in your target population, then the people distributing those bed nets will probably notice. There is also an established literature on “experimental methods” for these kinds of interventions that tells us that we need to be taking measurements and how to do so and how to interpret them.
By contrast, if you’re slightly reducing the odds of an AI catastrophe, it’s not immediately obvious or agreed-upon what observable changes this ought to produce in the real world, and a grant funder isn’t very likely to notice those changes unless they specifically go and look for them. They’re also less likely to specifically go and look for them in an effective way, because the literature on experimental methods for politics is much less well-developed than the literature on experimental methods for public health.
My work so far has mostly been about doing the advocacy, rather than establishing better metrics to evaluate the impact of that advocacy. That said, in posts 1 and 7 of this sequence, I do suggest some starting points. I encourage funders to look at figures like the number of meetings had with politicians, the number of events that draw in a significant number of politicians, the number of (positive) mentions in mainstream ‘earned media’, the number of endorsements that are included in Congressional offices’ press releases, and the number (and relative importance) of edits made to Congressional bills.
If your work is focused on the executive or judicial branch instead of on Congress, you could adapt some of those metrics accordingly, e.g., edits to pending regulation or executive orders, or citations to your amicus curiae briefs in judicial opinions, and so on.

Mass_Driver 19 Jun 2025 22:29 UTC
4 points
1
in reply to: 307th’s comment on: So You Want to Work at a Frontier AI Lab
> frontier labs are only pretending to try to solve alignment

>>This is probably the main driver of our disagreement.
I agree with your diagnosis! I think Sam Altman is a sociopathic liar, so the fact that he signed the statement on AI risk doesn’t convince me that he cares about alignment. I feel reasonably confident about that belief. Zvi’s series on Moral Mazes apply here: I don’t claim that you literally can’t mention existential risk at OpenAI, but if you show signs of being earnestly concerned enough about it to interefere with corporate goals, then I believe you’ll be sidelined.
I’m much less confident about whether or not successful alignment looks like normal deep learning work; I know more about corporate behavior than I do about technical AI safety. It seems odd and unlikely to me that the same kind of work (normal deep learning) that looks like it causes a series of major problems (power-seeking, black boxes, emergent goals) when you do a moderate amount of it would wind up solving all of those same problems when you do a lot of it, but I’m not enough of a technical expert to be sure that that’s wrong.
Because there are independent, non-technical reasons for people to want to believe that normal deep learning will solve alignment (it means they get to take fun, high-pay, high-status jobs at AI developers without feeling guilty about it), if you show me a random person who believes this and I don’t know anything about their incorruptiability or the clarity of their thinking ahead of time, then my prior is that most of the people in the random distribution that this person was drawn from probably arrived at the belief mostly out of convenience and temptation, rather than mostly by becoming technically convinced of the merits of a position that seems a priori unlikely to me. However, I can’t be sure—perhaps it’s more likely than I think that normal deep learning can solve alignment.

Mass_Driver 19 Jun 2025 21:04 UTC
4 points
1
in reply to: 307th’s comment on: So You Want to Work at a Frontier AI Lab
Well, I can’t change the headline; I’m just a commenter. However, I think the reason why “frontier labs will fail at alignment while nonprofits can succeed” is that frontier labs are only pretending to try to solve alignment—it’s not actually a serious goal of their leadership, and it’s not likely to get meaningful support in terms of compute, recruiting, data, or interdepartmental collaboration, and in fact the leadership will probably actively interfere with your work on a regular basis because the intermediate conclusions you’re reaching will get in the way of their profits and hurt their PR. In order to do useful superalignment research, I suspect you sometimes need to warn about or at least openly discuss the serious threats that are posed by increasingly advanced AI, but the business model of frontier labs depends on pretending that none of those threats are actually serious. By contrast, the main obstacle at a nonprofit is that they might not have much funding, but at least whatever funding they do have will be earnestly directed at supporting your team’s work.

Mass_Driver 18 Jun 2025 20:09 UTC
4 points
1
in reply to: 307th’s comment on: So You Want to Work at a Frontier AI Lab
I suspect Joe would agree with me that the current odds that AI developers solve superalignment are significantly less than 20%.
Even if we concede your estimate of 20% for the sake of argument, though, what price are you likely to pay for increasing the odds of success by 0.01%? Suppose that, given enough time, nonprofit alignment researchers would eventually solve superalignment with 80% odds. In order to increase, e.g., Anthropic’s odds of success by 0.01%, are you boosting Anthropic’s capabilities in a way that shortens timelines in a way that decreases the amount of time that the nonprofit alignment teams have to solve superalignment in a way that reduces their odds of success by at least 0.0025%? If so, you’ve done net harm. If not, why not? What about Joe’s arguments that most for-profit alignment work has at least some applicability to capabilities do you find unconvincing?