Wei Dai

Karma: 46,103

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 25 Apr 2026 20:11 UTC
4 points
0
in reply to: Tom Davidson’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems
>doing it when ai can significantly accelerate efforts to solve these problems
I just wrote a comment on EAF arguing for pausing now instead of, or in addition to, later, from the perspective of Forethought’s “Better Futures” (which is similar to Michael’s “post-alignment problems”). Maybe you can chime in there if you’re interested in discussing this?

Wei Dai 22 Apr 2026 0:11 UTC
LW: 3 AF: 2
0
AF
in reply to: Lukas Finnveden’s comment on: UDT shows that decision theory is more puzzling than ever
If so, there’s a sense in which the CDT agents aren’t benefitting anymore than they would if they were UDT agents who got a CDT sticker.
If the situation is reversed, CDT being in the majority and UDT in the minority, CDT would still defect and thus not give UDT an advantage, so CDT seems more “evolutionarily stable” than UDT. This can’t be replicated just by giving UDT agents CDT stickers, since UDT-with-CDT-stickers would cooperate if they’re in the majority.

Wei Dai 22 Apr 2026 0:00 UTC
4 points
0
in reply to: Tom Davidson’s comment on: Tom Davidson’s Shortform
I could imagine that some of the acausal trade crowd have thought long enough about the space of decision theories and their implications to conclude that acausal trade is a consequence of many plausible DTs and is very likely happening.
I’m not aware of anyone who has claimed this, and wouldn’t trust such claims anyway, since humans just aren’t that good at fully delineating the space of plausible solutions to some philosophical question. E.g. for the first half century of decision theory as a field, nobody thought that updatelessness might be worth investigating.
(In general i’d defer to someone who thinks about this more on how likely acausal trade is to happen overall!)
I wouldn’t defer a lot, because humans are also bad at finding flaws in their favorite philosophical ideas. Think of, e.g., theorists/proponents of Objectivism or Communism.
I’d be curious if you have other counter-cases in mind.
There are too many civilizations in the multiverse to simulate all of them. We can only do a sampling and then decide how to trade based on the statistical properties, but this gives an incentive to free-ride (i.e. getting the benefits of trade from other civilizations that did not specifically simulate us, without paying the costs), which may cause an overall breakdown in trade.

Wei Dai 21 Apr 2026 4:16 UTC
5 points
0
on: Automating philosophy if Timothy Williamson is correct
We have Solomonoff Induction as a formalization of science, and logic/set theory as formalization of math, which are both very far from perfect, but there’s nothing even remotely like them for philosophy. How to explain or fix this?
From my perspective this is because we have much better understandings of the nature of math and science (or natural and formal sciences, to use OP’s language) than we do of philosophy (or to put it another way, our philosophies or math and science are much more advanced than our philosophy of philosophy, aka metaphilosophy), and this allows us to:
1. Formalize (albeit imperfectly so far) math and science.
2. Make faster progress in math and science ourselves.
3. Improve AI competence in these areas.
If philosophy is just a kind of science, then how to explain why our understanding of its nature lags so much behind that of other sciences, or if Williamson denies this, how to explain why 1-3 above is working so much better for math and (natural) science relative to philosophy?

Wei Dai 19 Apr 2026 20:13 UTC
15 points
4
in reply to: Matrice Jacobine’s comment on: Wei Dai’s Shortform
What do you think “coordinate a pause” would have looked like in a pre-AI-race world?
1. be more humble/uncertain about personally, organizationally, or civilizationally solving AI alignment/safety
2. talk about alignment/safety research potentially failing and having to coordinate a pause
3. don’t make plans/claims about unilaterally (i.e., without widespread agreement/consent) building a world-changing Friendly AI or Pivotal Task AI, thereby normalizing the idea and spreading it to others (and try to discourage such plans in others)
4. start/encourage LW discussions about politics, social activism/movements, large-scale coordination, in theory and practice
5. think ahead about and encourage discussions about potential strategies and obstacles to coordinating a pause
6. try to convince prominent AI researchers of AI x-risks and get them to spread the message to the public earlier
7. gather resources (people, funding, connections, etc.) and build institutions for a potential pause effort
8. depending on how these efforts go, do one or more of:
9. 1. try to pre-emptively ban/regulate AI development
  2. start an international safe AGI/ASI project while banning others
  3. wait for an AI arms race to start and then oppose it from a well-prepared position

Wei Dai 19 Apr 2026 3:36 UTC
8 points
2
in reply to: 1a3orn’s comment on: Wei Dai’s Shortform
Paul Christiano didn’t have these beliefs/assumptions (and was prominent on LW), and more and more people in AI safety had Paul-like views as DL became more popular/successful through the 2010s (including safety people at DeepMind, founded in 2010, and OpenAI, founded in 2015), but still no significant Pause position or advocacy appeared until 2022.

Wei Dai 19 Apr 2026 1:31 UTC
3 points
1
in reply to: StoicVibes’s comment on: Current AIs seem pretty misaligned to me
I’m worried this comment might be from a bot, because the line “My point isn’t that we should hand off alignment to AI. ” does not seem to make logical sense here.
You are right to be suspicious. There are several indicators in the comment by StoicVibes that suggest it may be an LLM-generated response rather than a coherent contribution to the discussion:
The Hallucinated Argument: As you noted, the sentence “My point isn’t that we should hand off alignment to AI” makes no sense in context. StoicVibes had not made a previous point in this thread. This is a classic “hallucination” where a bot generates a defensive clarification for a stance it never actually took.
The “Agreement” Loop: The comment begins by saying “I agree with the distinction you’re drawing,” but then proceeds to simply restate Wei Dai’s points using slightly different synonyms (e.g., “quieter” instead of “less legible”). It adds zero new information or unique perspective.
The Summary-Style Tone: The final sentence (“It’s that ‘looking aligned’ and ‘being able to solve alignment’ are very different thresholds...”) reads like a concluding “moral of the story” from a summarization prompt rather than a natural part of a conversation between researchers.

Wei Dai 19 Apr 2026 1:12 UTC
4 points
1
in reply to: habryka’s comment on: Tom Davidson’s Shortform
I was reacting to the lack of hedging or explicit uncertainty in the first line “If we’re in a sim, it’s being used for acausal trade”. Possibly Tom didn’t mean to express very high confidence by it (I see that in a more formal occasion he does explicitly state his uncertainty about acausal trade^[1]), but I feel an obligation to err on the side of caution here, in case some readers do interpret it as expressing high confidence.
1. ^
  Quote from the link: Of course, unlike the causal case, whether consensus goods get funded depends on whether agents want to do acausal cooperation at all—which depends on their decision theories and their beliefs about their degree of correlation with others.

Wei Dai 18 Apr 2026 17:35 UTC
21 points
11
in reply to: Tom Davidson’s comment on: Tom Davidson’s Shortform
At this point we can’t be very sure that acausal trade is even a thing, since we don’t have a formal solution to decision theory we can be confident in (or even meet the lower bar of not having clearly serious flaws), and therefore can’t derive acausal trade from it as a solution to some decision problem. (Or have some other way to gain high justified confidence that acausal trade is actually a thing.)
And then even if acausal trade is a thing, whoever is running sims of us may not be running a decision theory that allows or recommends acausal trade, or we may not have anything to trade that they want, or the trade fails for some other reason like high “transaction costs”.

Wei Dai 18 Apr 2026 9:03 UTC
4 points
0
on: What’s actually inside 1,259 hours of AI safety podcasts?
This seems potentially very useful. Feature request: please put the date of the podcast episode on the search results.

Wei Dai 18 Apr 2026 8:38 UTC
23 points
2
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
Another perhaps interesting observation is that for the first ~20 years of AI safety (i.e., until “Death with Dignity” in 2022), the field was almost fully united behind this “try to build safe ASI, don’t bother trying to coordinate a pause” strategy.^[1] This seems remarkable both for itself (how to explain the lack of diversity of thought on this topic?) and for the lack of anyone noticing or investigating this anomaly until now.
1. ^
  Aside from my own posts, the closest talk or writing I can remember or find that’s in favor of a pause-like position before 2022 is Brian Tomasik’s 2013 International Cooperation vs. AI Arms Race but he wasn’t that enthusiastic about it: “I suspect more direct MIRI-type research has higher expected value, but among EAs who don’t want to fund MIRI specifically, encouraging donations toward international cooperation could be valuable, since it’s certainly a more mainstream cause.”
  Does anyone remember anything else? I tried several AI-powered queries/searches, but don’t fully trust the negative results since it’s hard to be sure what keywords to use, and maybe some materials have fallen off the indexed web.

Wei Dai 18 Apr 2026 1:24 UTC
69 points
7
on: Wei Dai’s Shortform
A historical note re: MIRI vs EA

When I was arguing against MIRI trying to build Friendly AI (now “aligned ASI”) circa 2012, I think 3 people at MIRI were thinking the most about strategy and occasionally responded to me: @lukeprog @CarlShulman @Eliezer Yudkowsky. Only one of them actually changed their mind about this (namely Eliezer), while the other two left MIRI and don’t support AI pause to this day, AFAIK. (Luke now “leads our grantmaking on AI governance and policy” at OpenPhil/Coefficient. Carl became an advisor to OpenPhil and is now Research Director at Situational Awareness LP.)
Seem significant (although I’m not sure of the implications) that the “try to build safe ASI, don’t bother trying to coordinate a pause” strategy originated at MIRI and only a minority of people there who defended it actually changed their mind. (It’s also curious how quiet both Luke and Carl are these days, at least in public.)
EDIT: To clarify, Carl does seem to support some kinds of short pauses under some future circumstances. (Quote: “To the extent you have a willingness to do a pause, it’s going to be much more impactful later on. And even worse, it’s possible that a pause, especially a voluntary pause, then is disproportionately giving up the opportunity to do pauses at that later stage when things are more important.”)
(@Connor Leahy I thought of this while reading your You can only build safe ASI if ASI is globally banned, which reminded me of my past arguments to MIRI.)

Wei Dai 16 Apr 2026 1:35 UTC
LW: 32 AF: 15
9
AF
on: Current AIs seem pretty misaligned to me
There are two distinct safety problems with handing off alignment research to AI:
1. Alignment: AI doesn’t try very hard, relative to other motivations like appearing to succeed.
2. Capabilities: AI just isn’t very good at solving hard alignment problems, making “innocent” mistakes despite trying hard.
It could be that the “capabilities problem” here is actually the more serious safety problem, due to being less legible. In other words, the alignment problem is fairly legible because its effects are apparent even in relatively easy tasks we assign to AI, which humans can fairly easily notice, as you and other people here and elsewhere are noticing. (I think this is already contributing a significant amount to the general anti-AI sentiment among the public, because many can see the apparent misalignment in their own AI usage.)
But suppose we solve this problem so that AIs seem pretty aligned, but they still can’t solve hard alignment problems reliably, only generate solutions that look good to humans and themselves...

Wei Dai 14 Apr 2026 20:17 UTC
3 points
1
in reply to: Lukas Finnveden’s comment on: No77e’s Shortform
If the society is “more capable (and good) than us” in all the important ways, then they’d also be better at making themselves smarter/wise than we would have been, and better at handling the transition, so further pauses really wouldn’t have contributed much.
Even this society, if it’s in the future, then part of the transition would have already occurred, so they won’t have the opportunity to make it go better. So by not pausing now we’d permanently give up this opportunity.
Take the issue in this recent comment, of building an initial AGI that reasons well or poorly about domains that lack fast/cheap feedback signals. It seems very plausible that our long-term civilizational trajectory is significantly affected by which type of AGI gets built first. Suppose we end up building one that reasons poorly about such domains, then:
1. The post-AGI civilization may end up being less capable (and good) than us on average, or in some important ways.
2. Even if they’re actually more capable (and good) than us in all the important ways, they could have been even better if only we had built an AGI that reasons well in such domains, but they can’t go back in time and change this.

Wei Dai 14 Apr 2026 5:04 UTC
5 points
0
on: AI for epistemics: the good, the bad and the ugly

AI will be able to think much more cheaply and quickly than humans. Partly this will mean that we can reach many more insights with much less effort. Partly this will make it possible to understand things that are currently infeasible for us to understand (because it would take too many humans too long to figure it out).

AI can ‘know’ much more than any human. Right now, a lot of information is siloed in specific expert communities, and it’s slow to filter out to other places even when it would be very useful there. AI will be able to port and apply knowledge much more quickly to the relevant places.

I think 5 or 10 years ago it was reasonable to hope for this (and some people explicitly did, e.g. Paul Christiano’s IDA assumed that you could train a base model to reason like a human and reach arbitrary intelligence/capabilities levels by iterating it enough times), but today we already have AIs that can think much more cheaply and quickly than humans, and know a lot more than any human, but outside of certain domains like coding and math (where RLVR is possible), this is proving surprisingly unhelpful. For example, I can’t take a post or paper in a field that I’m not familiar with, ask an AI to think a lot about it and tell me if its conclusions are reasonable or not, and get a trustworthy answer back. (And it seems like this has been improving at an imperceptible rate, if at all, while math/coding makes leaps and bounds.)

From my perspective the world is already going down my more pessimistic scenario, of AI differentially accelerating fields with fast/cheap feedback signals, like math, coding, manipulating humans (e.g. via sycophancy and plausible-sounding arguments), while not helping much with fields like philosophy, long-horizon strategy, even soft science fields like economics, that lack such signals. The default AI development trajectory at this point probably involves expanding RLVR training to all the fields where it’s feasible, like hard sciences, technology R&D, short-horizon agency, while other areas lag behind.

Do you agree with this view, or perhaps have your own explanation of why AI reasoning/knowledge has been of rather limited use so far, and why that might change before it’s too late? (Or perhaps think AI reasoning is generally more useful/helpful than it appears to me?)

I guess I do still have some hope that the current AI trajectory is actually a dead end. I.e., maybe RLVR won’t work so well outside of math/coding, and when we go back to the drawing board and come up with actual AGI, it will be able to reason in these low-feedback domains at least as well as humans. Are you mostly thinking along these lines?
What links here?
- Wei Dai's comment on No77e’s Shortform by No77e (14 Apr 2026 20:17 UTC; 3 points)

Wei Dai 14 Apr 2026 1:33 UTC
5 points
0
in reply to: Lukas Finnveden’s comment on: No77e’s Shortform
There’s just a decent chance that we do an ok job of developing AIs and handing over to a society that’s more capable than us at dealing with these issues (not a super high bar), in which case a pause wouldn’t add much.
The aim of a pause would be to plan out the transition better, or make humans smarter/wiser so they can navigate the transition better, so that we end up handing over remaining problems to a counterfactually more capable society. In other words, the bar shouldn’t be “more capable than us” but a society that could realistically be achieved with a pause.
The arguments that make me feel most pessimistic about the future are arguments that humans might just not be motivated to do good things — but it’s not clear why pauses would help much with that issue.
One issue related to this is that humans today largely want to do good things as a side effect of virtue signaling / status games that they’re doing/playing. This is currently far from optimal, which makes me scared to undergo an AI transition that could potentially lock-in such highly suboptimal motivations/values, and also scared that the AI transition could just scramble or reset these status games and remove what good motivations/values we do have. A pause would preserve the status quo and give people more time to think about such issues (including time for the idea to spread), and potentially find ways to make the AI transition go better in these regards (compared to today when there has been almost no thought on these issues at all).
But see also this recent quick take where I expressed that my optimism about a pause is pretty limited.

Wei Dai 12 Apr 2026 21:21 UTC
5 points
−7
in reply to: Lukas Finnveden’s comment on: No77e’s Shortform
Applying the idea is tricky and context-dependent. For example, gathering evidence for scheming seems unambiguously good, but actually solving scheming could be bad (unless you’re sure that such evidence can’t be gathered, or companies will not gate on this problem regardless), because some time in the future, it may well become legible enough to be gating deployment. (Also keep in mind that it’s not just legibility/gating by the companies, but also by other policymakers such as voters and politicians.)
Given the tradeoffs apparent to me (including that the benefits of solving scheming are limited by other safety problems), I think it may well be an example of a safety problem that is net negative to work on, and something I wouldn’t want to do myself. But I’m unsure how to argue for this convincingly (and also am just not certain enough to want to talk other people out of working on this specifically) which is why I’m only talking about it in response to your comment.

Wei Dai 12 Apr 2026 4:48 UTC
30 points
6
in reply to: No77e’s comment on: No77e’s Shortform
I respect this—all of our options are bad and unlikely to work, the situation is desperate, and I have no plan better than playing a portfolio of all the different desperate hard strategies in the hopes that one of them works.
I used to support such a portfolio approach, but subsequently realized that it’s actually not safe (i.e., is potentially net-negative even aside from opportunity costs), or the portfolio has be restricted a lot. This is because due to the existence of illegible AI safety problems, solving some (i.e., more legible) AI safety problems can actually make the overall situation worse, by increasing the chances of an unsafe AI being developed or deployed.
According to this logic, safer strategies include:
1. Pausing AI, and other actions that help broadly with both legible and illegible problems, like improving societal epistemic health.
2. Making illegible problems more legible.
3. Working directly on illegible problems.
Another reason to think that many “AI safety strategies” are actually not safe is that even nominally altruistic humans are more power/status-seeking^[1] than actually altruistic, and one way this manifests is that they tend to neglect risks more than they should (if they were actually altruistic). See my Managing risks while trying to do good. BTW these days I think not making this idea more prominent early in rationalism/EA/AI safety is a core failure that is upstream of many other errors.
1. ^
  I have an old post about power/status being fundamental to human motivation, which I remember @Scott Alexander liked.

Wei Dai 12 Apr 2026 3:46 UTC
4 points
1
in reply to: williawa’s comment on: Pausing AI Is the Best Answer to Post-Alignment Problems
But if you are a moral realist, and its meaningful to talk about making “moral errors”, i.e. there is a way to infer which values are “correct”, and there is a way to fall short of that, and this is a separate thing from making correct inferences about which actions are good wrt a set of predetermined values, then the ASI will not make such errors, because making correct inferences is a superintelligence’s whole schtick.
It’s not only moral realists who have to worry about moral errors. See #3 in my Six Plausible Meta-Ethical Alternatives:
There aren’t facts about what everyone should value, but there are facts about how to translate non-preferences (e.g., emotions, drives, fuzzy moral intuitions, circular preferences, non-consequentialist values, etc.) into preferences. These facts may include, for example, what is the right way to deal with ontological crises. The existence of such facts seems plausible because if there were facts about what is rational (which seems likely) but no facts about how to become rational, that would seem like a strange state of affairs.
Perhaps more importantly, ASI may lack philosophical competence, despite superhuman competence in other areas. It’s unclear why ASI must be philosophically competence, and seemingly reasons to suspect that they will not be. See my posts Some Thoughts on Metaphilosophy and AI doing philosophy = AI generating hands?

Wei Dai 12 Apr 2026 3:32 UTC
10 points
4
on: Pausing AI Is the Best Answer to Post-Alignment Problems
But there are many bad outcomes to avert, which makes pausing AI—as difficult as that would be—easier than solving all the post-alignment problems in a short time span.
I agree this is a good reason to support pausing AI, and suspect that more people would support pausing if they realized/understood this, but I’m not sure… You can see here where I gave a version of this argument to @wdmacaskill but did not get a response, and here where I reposted the same argument as a LW shortform, but it didn’t get much uptake (or pushback).
Research agendas on post-alignment problems rarely propose “pause/slow down AI development” as a mitigation.
Yeah, it generally confuses me why people who talk about post-alignment problems^[1] tend not to support pause, and people who support pause tend not to talk much about these problems.
1. ^
  Perhaps “non-alignment AI x-risks” would be more accurate to not imply that they are after alignment in some sense? Or maybe “ex-alignment problems”?