Wei Dai

Karma: 46,347

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 24 Jun 2026 4:38 UTC
13 points
7
on: Superintelligence vs. The Second Strike
AI can make a country much less sensitive to damage from a second strike, because AI infrastructure can be dispersed and any AIs that are “killed” during an attack can be restored from backup or from surviving copies once the war is over. Unlike humans, an AI could be fine with 99% destruction of their own side, as long as the enemy is 100% destroyed, since the surviving 1% could rebuild/restore and then take over the lightcone without opposition.

(Not sure if you overlooked this, or are deliberately limiting the scope of your analysis.)

Wei Dai 22 Jun 2026 7:14 UTC
6 points
1
in reply to: Mo Putera’s comment on: Wei Dai’s Shortform
Seems fine for @Noosphere89 to bring up Connor’s comment again. I (or any human) can easily overlook, misunderstand, or fail to pay enough attention to some important argument, and end up changing my mind just by reconsidering it.

Wei Dai 22 Jun 2026 7:03 UTC
15 points
2
in reply to: Noosphere89’s comment on: Wei Dai’s Shortform
One piece of advice on philosophy is that you should focus on specific, high impact problems and try to avoid conceptions of the problem that encompass everything. You should beware the urge to generalize problems so that you can try to have a solution in more cases, since there might just be no solution.
It seems like if somebody followed this advice, they wouldn’t have worked on philosophy of math (“you should be working on specific high impact math problems!”), philosophy of science, decision theory (“just think about specific high impact decisions!”), etc. (Do you think these fields are worth working on?)
I do agree with “there might just be no solution”, both in metaphilosophy as well as any of these other philosophical fields (which all still lack definitive solutions), but how would you establish that with any certainty unless a bunch of people tried hard to find solutions and failed (or alternatively tried to find arguments that a general solution isn’t possible, which would itself constitute a kind of solution)?
See also my Some Thoughts on Metaphilosophy, especially the section “Philosophy as meta problem solving” where I tried to explain in a more positive way why the “urge to generalize problems” can be a good idea.

Wei Dai 21 Jun 2026 22:16 UTC
7 points
0
in reply to: StanislavKrym’s comment on: Wei Dai’s Shortform
This is a miscommunication. I meant I couldn’t find an objection to the ending of the show based on that it’s a bad representation of a philosopher, or teaches a wrong lesson about the value of philosophy.

Wei Dai 21 Jun 2026 21:29 UTC
87 points
0
on: Wei Dai’s Shortform
The first podcast episode I’ve ever participated in has been released, if anyone wants to get an update on what I’ve been thinking about recently, in audio form. (A transcript is also available.) Thanks to Fin Moorhouse @fin for the conversation and handling the logistics/production.

BTW, related to the theme of humans being bad at philosophy and also just not caring very much about it, I recently finished The Good Place (spoiler alert), and it bothered me a lot that it ends with Chidi, the philosopher character, deciding to end his (after)life instead of using the infinite time and resources available to solve philosophy, or to participate in such a project.

Worse, I can’t find any existing objection to this, including from actual philosophers, who mostly seem happy just to have any representation at all in pop culture.
Meanwhile in AI safety, our best plan is still to defer to humans on questions like “What should ASI do with our lightcone?”

Wei Dai 28 Apr 2026 10:50 UTC
24 points
3
on: LessWrong Shows You Social Signals Before the Comment
To me, the most valuable part of LessWrong was how it encouraged interesting contrarian comments. Many of us value truth-seeking, so I hope users and moderators optimize the website to better reflect that value.
Seeing a high-karma post or comment tends to make me more motivated to find a serious flaw in it (such as ignoring or overlooking a strong counterargument), since that would net a higher status/karma reward than finding a similar flaw in a low-karma content.

Wei Dai 25 Apr 2026 20:11 UTC
4 points
0
in reply to: Tom Davidson’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems
>doing it when ai can significantly accelerate efforts to solve these problems
I just wrote a comment on EAF arguing for pausing now instead of, or in addition to, later, from the perspective of Forethought’s “Better Futures” (which is similar to Michael’s “post-alignment problems”). Maybe you can chime in there if you’re interested in discussing this?

Wei Dai 22 Apr 2026 0:11 UTC
LW: 3 AF: 2
0
AF
in reply to: Lukas Finnveden’s comment on: UDT shows that decision theory is more puzzling than ever
If so, there’s a sense in which the CDT agents aren’t benefitting anymore than they would if they were UDT agents who got a CDT sticker.
If the situation is reversed, CDT being in the majority and UDT in the minority, CDT would still defect and thus not give UDT an advantage, so CDT seems more “evolutionarily stable” than UDT. This can’t be replicated just by giving UDT agents CDT stickers, since UDT-with-CDT-stickers would cooperate if they’re in the majority.

Wei Dai 22 Apr 2026 0:00 UTC
4 points
0
in reply to: Tom Davidson’s comment on: Tom Davidson’s Shortform
I could imagine that some of the acausal trade crowd have thought long enough about the space of decision theories and their implications to conclude that acausal trade is a consequence of many plausible DTs and is very likely happening.
I’m not aware of anyone who has claimed this, and wouldn’t trust such claims anyway, since humans just aren’t that good at fully delineating the space of plausible solutions to some philosophical question. E.g. for the first half century of decision theory as a field, nobody thought that updatelessness might be worth investigating.
(In general i’d defer to someone who thinks about this more on how likely acausal trade is to happen overall!)
I wouldn’t defer a lot, because humans are also bad at finding flaws in their favorite philosophical ideas. Think of, e.g., theorists/proponents of Objectivism or Communism.
I’d be curious if you have other counter-cases in mind.
There are too many civilizations in the multiverse to simulate all of them. We can only do a sampling and then decide how to trade based on the statistical properties, but this gives an incentive to free-ride (i.e. getting the benefits of trade from other civilizations that did not specifically simulate us, without paying the costs), which may cause an overall breakdown in trade.

Wei Dai 21 Apr 2026 4:16 UTC
5 points
0
on: Automating philosophy if Timothy Williamson is correct
We have Solomonoff Induction as a formalization of science, and logic/set theory as formalization of math, which are both very far from perfect, but there’s nothing even remotely like them for philosophy. How to explain or fix this?
From my perspective this is because we have much better understandings of the nature of math and science (or natural and formal sciences, to use OP’s language) than we do of philosophy (or to put it another way, our philosophies or math and science are much more advanced than our philosophy of philosophy, aka metaphilosophy), and this allows us to:
1. Formalize (albeit imperfectly so far) math and science.
2. Make faster progress in math and science ourselves.
3. Improve AI competence in these areas.
If philosophy is just a kind of science, then how to explain why our understanding of its nature lags so much behind that of other sciences, or if Williamson denies this, how to explain why 1-3 above is working so much better for math and (natural) science relative to philosophy?

Wei Dai 19 Apr 2026 20:13 UTC
15 points
4
in reply to: Matrice Jacobine’s comment on: Wei Dai’s Shortform
What do you think “coordinate a pause” would have looked like in a pre-AI-race world?
1. be more humble/uncertain about personally, organizationally, or civilizationally solving AI alignment/safety
2. talk about alignment/safety research potentially failing and having to coordinate a pause
3. don’t make plans/claims about unilaterally (i.e., without widespread agreement/consent) building a world-changing Friendly AI or Pivotal Task AI, thereby normalizing the idea and spreading it to others (and try to discourage such plans in others)
4. start/encourage LW discussions about politics, social activism/movements, large-scale coordination, in theory and practice
5. think ahead about and encourage discussions about potential strategies and obstacles to coordinating a pause
6. try to convince prominent AI researchers of AI x-risks and get them to spread the message to the public earlier
7. gather resources (people, funding, connections, etc.) and build institutions for a potential pause effort
8. depending on how these efforts go, do one or more of:
9. 1. try to pre-emptively ban/regulate AI development
  2. start an international safe AGI/ASI project while banning others
  3. wait for an AI arms race to start and then oppose it from a well-prepared position

Wei Dai 19 Apr 2026 3:36 UTC
8 points
2
in reply to: 1a3orn’s comment on: Wei Dai’s Shortform
Paul Christiano didn’t have these beliefs/assumptions (and was prominent on LW), and more and more people in AI safety had Paul-like views as DL became more popular/successful through the 2010s (including safety people at DeepMind, founded in 2010, and OpenAI, founded in 2015), but still no significant Pause position or advocacy appeared until 2022.

Wei Dai 19 Apr 2026 1:31 UTC
3 points
1
in reply to: StoicVibes’s comment on: Current AIs seem pretty misaligned to me
I’m worried this comment might be from a bot, because the line “My point isn’t that we should hand off alignment to AI. ” does not seem to make logical sense here.
You are right to be suspicious. There are several indicators in the comment by StoicVibes that suggest it may be an LLM-generated response rather than a coherent contribution to the discussion:
The Hallucinated Argument: As you noted, the sentence “My point isn’t that we should hand off alignment to AI” makes no sense in context. StoicVibes had not made a previous point in this thread. This is a classic “hallucination” where a bot generates a defensive clarification for a stance it never actually took.
The “Agreement” Loop: The comment begins by saying “I agree with the distinction you’re drawing,” but then proceeds to simply restate Wei Dai’s points using slightly different synonyms (e.g., “quieter” instead of “less legible”). It adds zero new information or unique perspective.
The Summary-Style Tone: The final sentence (“It’s that ‘looking aligned’ and ‘being able to solve alignment’ are very different thresholds...”) reads like a concluding “moral of the story” from a summarization prompt rather than a natural part of a conversation between researchers.

Wei Dai 19 Apr 2026 1:12 UTC
5 points
1
in reply to: habryka’s comment on: Tom Davidson’s Shortform
I was reacting to the lack of hedging or explicit uncertainty in the first line “If we’re in a sim, it’s being used for acausal trade”. Possibly Tom didn’t mean to express very high confidence by it (I see that in a more formal occasion he does explicitly state his uncertainty about acausal trade^[1]), but I feel an obligation to err on the side of caution here, in case some readers do interpret it as expressing high confidence.
1. ^
  Quote from the link: Of course, unlike the causal case, whether consensus goods get funded depends on whether agents want to do acausal cooperation at all—which depends on their decision theories and their beliefs about their degree of correlation with others.

Wei Dai 18 Apr 2026 17:35 UTC
21 points
11
in reply to: Tom Davidson’s comment on: Tom Davidson’s Shortform
At this point we can’t be very sure that acausal trade is even a thing, since we don’t have a formal solution to decision theory we can be confident in (or even meet the lower bar of not having clearly serious flaws), and therefore can’t derive acausal trade from it as a solution to some decision problem. (Or have some other way to gain high justified confidence that acausal trade is actually a thing.)
And then even if acausal trade is a thing, whoever is running sims of us may not be running a decision theory that allows or recommends acausal trade, or we may not have anything to trade that they want, or the trade fails for some other reason like high “transaction costs”.

Wei Dai 18 Apr 2026 9:03 UTC
4 points
0
on: What’s actually inside 1,259 hours of AI safety podcasts?
This seems potentially very useful. Feature request: please put the date of the podcast episode on the search results.

Wei Dai 18 Apr 2026 8:38 UTC
23 points
2
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
Another perhaps interesting observation is that for the first ~20 years of AI safety (i.e., until “Death with Dignity” in 2022), the field was almost fully united behind this “try to build safe ASI, don’t bother trying to coordinate a pause” strategy.^[1] This seems remarkable both for itself (how to explain the lack of diversity of thought on this topic?) and for the lack of anyone noticing or investigating this anomaly until now.
1. ^
  Aside from my own posts, the closest talk or writing I can remember or find that’s in favor of a pause-like position before 2022 is Brian Tomasik’s 2013 International Cooperation vs. AI Arms Race but he wasn’t that enthusiastic about it: “I suspect more direct MIRI-type research has higher expected value, but among EAs who don’t want to fund MIRI specifically, encouraging donations toward international cooperation could be valuable, since it’s certainly a more mainstream cause.”
  Does anyone remember anything else? I tried several AI-powered queries/searches, but don’t fully trust the negative results since it’s hard to be sure what keywords to use, and maybe some materials have fallen off the indexed web.

Wei Dai 18 Apr 2026 1:24 UTC
69 points
7
on: Wei Dai’s Shortform
A historical note re: MIRI vs EA

When I was arguing against MIRI trying to build Friendly AI (now “aligned ASI”) circa 2012, I think 3 people at MIRI were thinking the most about strategy and occasionally responded to me: @lukeprog @CarlShulman @Eliezer Yudkowsky. Only one of them actually changed their mind about this (namely Eliezer), while the other two left MIRI and don’t support AI pause to this day, AFAIK. (Luke now “leads our grantmaking on AI governance and policy” at OpenPhil/Coefficient. Carl became an advisor to OpenPhil and is now Research Director at Situational Awareness LP.)
Seem significant (although I’m not sure of the implications) that the “try to build safe ASI, don’t bother trying to coordinate a pause” strategy originated at MIRI and only a minority of people there who defended it actually changed their mind. (It’s also curious how quiet both Luke and Carl are these days, at least in public.)
EDIT: To clarify, Carl does seem to support some kinds of short pauses under some future circumstances. (Quote: “To the extent you have a willingness to do a pause, it’s going to be much more impactful later on. And even worse, it’s possible that a pause, especially a voluntary pause, then is disproportionately giving up the opportunity to do pauses at that later stage when things are more important.”)
(@Connor Leahy I thought of this while reading your You can only build safe ASI if ASI is globally banned, which reminded me of my past arguments to MIRI.)

Wei Dai 16 Apr 2026 1:35 UTC
LW: 32 AF: 15
9
AF
on: Current AIs seem pretty misaligned to me
There are two distinct safety problems with handing off alignment research to AI:
1. Alignment: AI doesn’t try very hard, relative to other motivations like appearing to succeed.
2. Capabilities: AI just isn’t very good at solving hard alignment problems, making “innocent” mistakes despite trying hard.
It could be that the “capabilities problem” here is actually the more serious safety problem, due to being less legible. In other words, the alignment problem is fairly legible because its effects are apparent even in relatively easy tasks we assign to AI, which humans can fairly easily notice, as you and other people here and elsewhere are noticing. (I think this is already contributing a significant amount to the general anti-AI sentiment among the public, because many can see the apparent misalignment in their own AI usage.)
But suppose we solve this problem so that AIs seem pretty aligned, but they still can’t solve hard alignment problems reliably, only generate solutions that look good to humans and themselves...

Wei Dai 14 Apr 2026 20:17 UTC
3 points
1
in reply to: Lukas Finnveden’s comment on: No77e’s Shortform
If the society is “more capable (and good) than us” in all the important ways, then they’d also be better at making themselves smarter/wise than we would have been, and better at handling the transition, so further pauses really wouldn’t have contributed much.
Even this society, if it’s in the future, then part of the transition would have already occurred, so they won’t have the opportunity to make it go better. So by not pausing now we’d permanently give up this opportunity.
Take the issue in this recent comment, of building an initial AGI that reasons well or poorly about domains that lack fast/cheap feedback signals. It seems very plausible that our long-term civilizational trajectory is significantly affected by which type of AGI gets built first. Suppose we end up building one that reasons poorly about such domains, then:
1. The post-AGI civilization may end up being less capable (and good) than us on average, or in some important ways.
2. Even if they’re actually more capable (and good) than us in all the important ways, they could have been even better if only we had built an AGI that reasons well in such domains, but they can’t go back in time and change this.