Wei Dai

Karma: 47,137

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 4 Aug 2026 8:13 UTC
LW: 2 AF: 2
0
AF
on: Notes on technical alignment via human-like social drives
I don’t think that theory [social status] explains quite as much as those people think it does
Do you have a link/explanation for this? I think this may be fairly cruxy, because I’m guessing your intuitions for truth-seeking disagreeable nerd AGI are substantially based on truth-seeking disagreeable nerd humans, so it matters what those humans’ real motivations are.
One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn’t have its own status motivations or other long-term goals?

Wei Dai 4 Aug 2026 7:35 UTC
2 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I posted a screenshot of this shortform to X on a whim^[1] and it ended up getting 238k views, 1.1k likes, and increased my follower count from ~9k to ~13k. One observation is unlike when I started tweeting ~3 years ago, there was no pushback from e/acc, which seems to have completely disappeared from X.
1. ^
  thought I could maybe catch some algorithmic coattails from the “situational awareness” situation

Wei Dai 3 Aug 2026 23:16 UTC
LW: 5 AF: 4
0
AF
in reply to: Max Harms’s comment on: A Conflict Between AI Alignment and Philosophical Competence
I don’t think there’s any tension for the frames of corrigibility that I prefer, where the corrigible agent terminally-values having a certain kind of relationship with the principal.
A philosophically competent agent would seemingly (until it fully solved morality/axiology) have uncertainty about any terminal values, including “having a certain kind of relationship with the principal”. So I don’t understand how your approach or framing gets away from my critique in the OP. (Would be interested in having a chat about this if that might help.)

Wei Dai 31 Jul 2026 9:12 UTC
3 points
0
in reply to: Ondřej_Kubů’s comment on: The Long (Self-)Correction
Thanks for the pointer, which I didn’t know about, but yeah I mostly mean discussing/analyzing the relationship between status seeking and altruism, strategizing how to best take advantage of it, including how to avoid problems caused by misalignment between the two (like in this example), and not just going ahead and pulling the lever as you put it.

Wei Dai 29 Jul 2026 19:25 UTC
6 points
0
in reply to: habryka’s comment on: Some Dating Stories
That was my memory/assessment, but of course I’d be biased in that direction. I guess the only evidence I can easily point out (for a large role in things dying down) is that Alicorn was tracking all of the top-level posts on the topic, and nothing came after mine. I’d be interested in anyone else’s take, if they were around at the time and remember this drama. I guess @Roko should remember, since it he was centrally involved (and wrote the comment that triggered the main debate).
What about the other posts that Alicorn tracked? Do you think the old debate was reasonably complete, or have additional thoughts/arguments that none of them brought up?

Wei Dai 29 Jul 2026 10:48 UTC
5 points
0
on: You can, in fact, bamboozle an unaligned AI into sparing your life
Are we in a simulation? What should we do?
I missed your call for links earlier. I’ve written a couple of pieces about this:
- Beyond Astronomical Waste (in short, try harder to ensure philosophical competence of future AIs or humans or civilization as a whole)
- https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=NrtKECeYASwoBJbki (“thinking and writing with potential simulators in mind as an additional audience, hoping my ideas might benefit or interest some of them even if they end up largely ignored in this reality”)

Wei Dai 28 Jul 2026 19:00 UTC
5 points
0
in reply to: habryka’s comment on: Some Dating Stories
Have you seen The Nature of Offense? I remember feeling very proud of this post and thinking that it kind of “saved” LW from the PUA war at the time. (It came last in a sequence of posts debating the controversy by many prominent posters, and got the highest karma by a large margin. Afterwards, the debate mostly died down, and people seemed to stop posting PUA content voluntarily.) @Ben Pace

Wei Dai 28 Jul 2026 8:31 UTC
3 points
0
in reply to: WillPetillo’s comment on: The Long (Self-)Correction
An acknowledged scope limitation is not the same as a flaw, and I’m not sure which of these you mean by “problem.” The explicit purpose of an AI Pause is to create time for some intentionally unspecified longer-term solution, on the grounds that there are many competing theories as to what such a solution looks like, whereas needing time to implement—and decide between them—is a common factor.
Good point/question. My intended meaning is closer to the former, and basically the “problem” is referring to the fact that I lacked a good handle for the concept that I often want to invoke, with “AI Pause” and “Long Reflection” being the closest ones. I think what you say here makes sense and I don’t intend for my new handle to replace “AI Pause” where “AI Pause” is more appropriate.
The rest of your comment seems like an interesting idea for solving part of the challenge, worth keeping in mind, and fleshing out and discussing in the future, when we have this “generalized externalized cost tax” that we can build your idea into.

Wei Dai 28 Jul 2026 8:01 UTC
7 points
0
in reply to: Toby_Ord’s comment on: The Long (Self-)Correction
Thanks, Toby, I highly appreciate your perspective. I’m honestly a bit surprised by our apparent convergence in views here, given that this part of your book sounds a lot more positive/optimistic than my post and you didn’t talk about most of the flaws I listed, except something like my #1: “The study of the positive is at a much earlier stage of development.” I’m curious if this was a matter of choice of emphasis or presentation, or if your views have substantively changed since writing the book.
Perhaps another potential difference between us is that I see existential security as being secured or securable only after the Long Self-Correction has largely finished (instead of as a precondition for the Long Reflection as you wrote in your book), because even if we secured ourselves against all other risks, humanity’s flaws imply that in the early stages there is seemingly no way to ensure that the process doesn’t go off-rails, locking us into a basin of attraction that ends up in a wrong destination (according to true normativity or actually competent reflection), or we just choose to end the process early out of collective overconfidence. (Both of these could constitute an existential risk in the sense of “permanently destroy humanity’s future potential”.)
Finally, I wonder if you can clear up something I’ve always wondered about the Long Reflection—was it meant to be pre-ASI, post-ASI, or agnostic about this? (See this comment for additional context.)

Wei Dai 25 Jul 2026 21:45 UTC
3 points
0
in reply to: DanB’s comment on: The Long (Self-)Correction
Yeah, I talked about this somewhat in this post, and this shortform.

Wei Dai 25 Jul 2026 21:37 UTC
3 points
0
in reply to: Ben Pace’s comment on: The Long (Self-)Correction
Let us not call our era the age of reason, or the enlightenment, for that is too grand; let us call it the age of getting-slightly-better.
The Long Self-Correction seems appropriately grand to me. :) Age of Reason and the Enlightenment seem to be instances of the very thing I criticize in the OP, “badly calibrated about our philosophical and strategic competence”. Have you tried reading some of the philosophy from those times?

Wei Dai 25 Jul 2026 21:29 UTC
9 points
3
in reply to: Chris Lakin’s comment on: The Long (Self-)Correction
I originally meant to propose a new name/idea for intellectual discussion among rationalists/EAs (where things like Orwellian connotations aren’t as important as the literal/logical meanings), but given that it has a chance of spreading further I suppose PR considerations should also be taken into account. But contra @bits I do want the name to be somewhat “negative”, i.e., suggest that we’re starting from a very flawed state, should be very wary of doing anything highly consequential in our current state, and whatever process we undertake has a very real chance of failure.

Wei Dai 25 Jul 2026 20:41 UTC
LW: 6 AF: 3
1
AF
in reply to: cousin_it’s comment on: The Long (Self-)Correction
Is this question philosophically meaningful?
Assuming you mean something like “Can philosophy answer this question?” I think “Maybe, but we probably won’t know until we do a lot more philosophy.” To put it another way, I think it’s very plausible (but far from certain) that we can eventually answer questions like this one, given enough competent reflection, and I want to make sure we definitively find out before we make any irreversible choices based on what we think our values are.

Wei Dai 25 Jul 2026 20:26 UTC
6 points
0
in reply to: bits’s comment on: The Long (Self-)Correction
Yeah, viatopia is another idea in the same cluster, but it’s explicitly framed as post-ASI:
Yet almost no one has articulated a positive vision for what comes after superintelligence. Few people are even asking, “What if we succeed?” Even fewer have tried to answer.
Same post also says it’s meant to be a generalization of Long Reflection:
Viatopia is a more general concept: the long reflection is one proposal for what viatopia would look like, but it need not be the only one.
However my memory says that the Long Reflection was meant to be pre-ASI, and looking back at where it was originally proposed (Toby Ord’s The Precipice), that still seems like the most plausible interpretation, although it wasn’t fully explicit about this. (Note that the Long Self-Correction is also meant to be pre-ASI, since I think humans are too flawed/unsafe to try to build ASI.) Quoting from Toby’s book (bolding added by me):
How can humanity have the greatest chance of achieving its potential? I think that at the highest level we should adopt a strategy proceeding in three phases:2
1. Reaching Existential Security
2. The Long Reflection
3. Achieving Our Potential
On this view, the first great task for humanity is to reach a place of safety—a place where existential risk is low and stays low. I call this existential security.
It has two strands. Most obviously, we need to preserve humanity’s potential, extracting ourselves from immediate danger so we don’t fail before we’ve got our house in order. This includes direct work on the most pressing existential risks and risk factors, as well as near-term changes to our norms and institutions.
But we also need to protect humanity’s potential—to establish lasting safeguards that will defend humanity from dangers over the longterm future, so that it becomes almost impossible to fail.3 Where preserving our potential is akin to fighting the latest fire, protecting our potential is making changes to ensure that fire will never again pose a serious threat.4 This will involve major changes to our norms and institutions (giving humanity the prudence and patience we need), as well as ways of increasing our general resilience to catastrophe. This needn’t require foreseeing all future risks right now. It is enough if we can set humanity firmly on a course where we will be taking the new risks seriously: managing them successfully right from their onset or sidestepping them entirely.
Note that existential security doesn’t require the risk to be brought down to zero. That would be an impossible target, and attempts to achieve it may well be counter-productive. What humanity needs to do is bring this century’s risk down to a very low level, then keep gradually reducing it from there as the centuries go on. In this way, even though there may always remain some risk in each century, the total risk over our entire future can be kept small.5 We could view this as a form of existential sustainability. Futures in which accumulated existential risk is allowed to climb toward 100 percent are unsustainable. So we need to set a strict risk budget over our entire future, parceling out this non-renewable resource with great care over the generations to come.
Ultimately, existential security is about reducing total existential risk by as many percentage points as possible. Preserving our potential is helping lower the portion of the total risk that we face in the next few decades, while protecting our potential is helping lower the portion that comes over the longer run. We can work on these strands in parallel, devoting some of our efforts to reducing imminent risks and some to building the capacities, institutions, wisdom and will to ensure that future risks are minimal.6
A key insight motivating existential security is that there appear to be no major obstacles to humanity lasting an extremely long time, if only that were a key global priority. As we saw in Chapter 3, we have ample time to protect ourselves against natural risks: even if it took us millennia to resolve the threats from asteroids, supervolcanism and supernovae, we would incur less than one percentage point of total risk.
The greater risk (and tighter deadline) stems from the anthropogenic threats. But being of humanity’s own making, they are also within our control. Were we sufficiently patient, prudent and coordinated, we could simply stop imposing such risks upon ourselves. We would factor in the hidden costs of carbon emissions (or nuclear weapons) and realize they are not a good deal. We would adopt a more mature attitude to the most radical new technologies—devoting at least as much of humanity’s brilliance to forethought and governance as to technological development.
In the past, the survival of humanity didn’t require much conscious effort: our past was brief enough to evade the natural threats and our power too limited to produce anthropogenic threats. But now our longterm survival requires a deliberate choice to survive. As more and more people come to realize this, we can make this choice. There will be great challenges in getting people to look far enough ahead and to see beyond the parochial conflicts of the day. But the logic is clear and the moral arguments powerful. It can be done.
If we achieve existential security, we will have room to breathe. With humanity’s longterm potential secured, we will be past the Precipice, free to contemplate the range of futures that lie open before us. And we will be able to take our time to reflect upon what we truly desire; upon which of these visions for humanity would be the best realization of our potential. We shall call this the Long Reflection.7
@Toby_Ord @wdmacaskill in case they want to weigh in on this.
What links here?
- Wei Dai's comment on The Long (Self-)Correction by Wei Dai (28 Jul 2026 8:01 UTC; 7 points)

Wei Dai 24 Jul 2026 6:26 UTC
13 points
7
in reply to: alkjash’s comment on: Mathematicians are Feeling the Doom
Note that I never used “highly strategically competent”! If we did have a group of highly strategically competent people, then what you say becomes a lot more plausible, but even the early AI risk people are far from that. Consider for example that a strategic consideration as obvious (in retrospect) as Legible vs. Illegible AI Safety Problems was written down only in late 2025.

Wei Dai 24 Jul 2026 4:01 UTC
35 points
24
on: Mathematicians are Feeling the Doom
Seems like the fundamental problem is that humanity’s strategic competence is extremely inadequate relative to its technical capabilities, and this (limited) strategic competence was front-loaded into AI risk (the most strategically competent people foresaw the problem and went into the field early, before AI was even a thing for everyone else). As AI risk awareness grows, the average strategic competence goes down as more and more people enter the field. (Consider the gap between someone who “are going to OpenAI without even having considered x-risk for five minutes”, and what’s actually needed to make a positive contribution or avoid making a negative contribution.)

Wei Dai 22 Jul 2026 18:03 UTC
5 points
0
in reply to: habryka’s comment on: Wei Dai’s Shortform
Hi Oliver, any updates on this (fixing the issue with Lightcone governance)?

Wei Dai 21 Jul 2026 20:38 UTC
20 points
5
on: Stop doing decision theory without metaphysics
1. My first post about UDT, Towards a New Decision Theory, was pretty explicit about its metaphysical assumptions, i.e., that the decision maker is a program that cares about what happens in other programs or mathematical structures, with Tegmark level IV explicitly cited: “I suggest that the Level 4 Multiverse should be considered the default setting for a general decision theory, since we cannot rule out the possibility that all mathematical structures do indeed exist physically, or that we have direct preferences on mathematical structures (in which case there is no need for them to exist ‘physically’).”
2. I guess FDT’s presentation did deemphasize this aspect, which seems good to point out (I don’t recall noticing it earlier, e.g., when the authors asked me to review the paper draft).
3. The increasing prominence of FDT relative to UDT over time (which you also point out here, and which I previously hadn’t paid much attention to) seems more sociopolitical than based on the merits, which is kind of disconcerting. Hopefully someone will correct me if I’m wrong, but I don’t know of any strong arguments why FDT should be preferred over UDT (the published papers didn’t try to argue this or talk about UDT at all, IIRC), and instead it seems like it has gained attention/mindshare mainly due to being published in an academic paper and Eliezer’s preference of it over UDT (i.e., preferring logical counterfactuals over logical conditionals), and then snowballing from there.
  1. I guess I didn’t object or pay much attention because I was expecting fairly rapid progress in decision theory so it didn’t matter much which version got published, and the important thing was just to draw more attention to this line of thought to help push the progress.
  2. Given that progress is now significantly stalled, it has become more of a signaling/reputation issue, where “rationalist decision theory” is used to judge the philosophical/intellectual competence of the rationalist community, in which case giving so much more prominence for FDT over UDT (instead of presenting them as roughly equal) seems counterproductive when some people prefer UDT over FDT.
4. Part of the issue is that it’s pretty unclear what metaphysics makes sense when thinking about logical uncertainty, logical counterfactuals, and logical updatelessness. I.e., unlike empirical updatelessness which fits very well with Platonism or Modal Realism (i.e., all possible worlds exist so why update (that is, assigning 0 weight to worlds you don’t observe and then renormalize)), it seemingly makes no sense to think that all impossible possible worlds exist. (Note that UDT 1.x is actually updateful regarding logical uncertainty, which UDT 2 tried to fix but in a not very satisfying way.)

Wei Dai 18 Jul 2026 22:08 UTC
19 points
10
in reply to: MichaelDickens’s comment on: Jai’s Shortform
These weakly-superhuman AIs would need to convince the humans to stop, not just refuse to help, otherwise it seems easy to train such refusals away.

Wei Dai 18 Jul 2026 5:33 UTC
LW: 20 AF: 6
11
AF
on: AI 2040: Plan A
I like that the space governance plan supplement acknowledges that defining torture and slavery are hard problems that ASI could potentially help with:
It might take non-trivial reflection by advanced ASI to figure out the precise bounds of this.
And in the sidenote it gives examples of difficult subproblems or dependencies that would need solved in order to do this, like “What exactly are negatively-valenced experiences?” But I have a couple of problems with the rest of the scenario given this:
1. If there are ASI that are philosophically competent enough to solve these particular problems in this timeframe, wouldn’t they likely also be competent enough to solve all or most other important philosophical problems, given just slightly more calendar time? Why don’t the delegates use their ASI assistants to solve all of morality and philosophy (or specifically answer “What does the ideal intergalactic civilization look like, given humanity as the starting point?”), and then negotiate the cosmic treaty?
2. AFAICT (it’s a bit hard to search the site, due to many collapsed sections), there is no mention in the rest of the plan how the ASI got to be philosophically competent, making it seem like such competence comes with superintelligence for free. I’m pretty sure at least a couple of the authors are aware of and agree (to some extent) with my concern that future AIs may not be philosophically competent by default (and one of my posts laying out this concern was even linked in another supplement), so maybe it’s just an oversight?