Wei Dai

Karma: 46,958

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 29 Jul 2026 19:25 UTC
3 points
0
in reply to: habryka’s comment on: Some Dating Stories
That was my memory/assessment, but of course I’d be biased in that direction. I guess the only evidence I can easily point out (for a large role in things dying down) is that Alicorn was tracking all of the top-level posts on the topic, and nothing came after mine. I’d be interested in anyone else’s take, if they were around at the time and remember this drama. I guess @Roko should remember, since it he was centrally involved (and wrote the comment that triggered the main debate).
What about the other posts that Alicorn tracked? Do you think the old debate was reasonably complete, or have additional thoughts/arguments that none of them brought up?

Wei Dai 29 Jul 2026 10:48 UTC
4 points
0
on: You can, in fact, bamboozle an unaligned AI into sparing your life
Are we in a simulation? What should we do?
I missed your call for links earlier. I’ve written a couple of pieces about this:
- Beyond Astronomical Waste (in short, try harder to ensure philosophical competence of future AIs or humans or civilization as a whole)
- https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=NrtKECeYASwoBJbki (“thinking and writing with potential simulators in mind as an additional audience, hoping my ideas might benefit or interest some of them even if they end up largely ignored in this reality”)

Wei Dai 28 Jul 2026 19:00 UTC
2 points
0
in reply to: habryka’s comment on: Some Dating Stories
Have you seen The Nature of Offense? I remember feeling very proud of this post and thinking that it kind of “saved” LW from the PUA war at the time. (It came last in a sequence of posts debating the controversy by many prominent posters, and got the highest karma by a large margin. Afterwards, the debate mostly died down, and people seemed to stop posting PUA content voluntarily.) @Ben Pace

Wei Dai 28 Jul 2026 8:31 UTC
2 points
0
in reply to: WillPetillo’s comment on: The Long (Self-)Correction
An acknowledged scope limitation is not the same as a flaw, and I’m not sure which of these you mean by “problem.” The explicit purpose of an AI Pause is to create time for some intentionally unspecified longer-term solution, on the grounds that there are many competing theories as to what such a solution looks like, whereas needing time to implement—and decide between them—is a common factor.
Good point/question. My intended meaning is closer to the former, and basically the “problem” is referring to the fact that I lacked a good handle for the concept that I often want to invoke, with “AI Pause” and “Long Reflection” being the closest ones. I think what you say here makes sense and I don’t intend for my new handle to replace “AI Pause” where “AI Pause” is more appropriate.
The rest of your comment seems like an interesting idea for solving part of the challenge, worth keeping in mind, and fleshing out and discussing in the future, when we have this “generalized externalized cost tax” that we can build your idea into.

Wei Dai 28 Jul 2026 8:01 UTC
4 points
0
in reply to: Toby_Ord’s comment on: The Long (Self-)Correction
Thanks, Toby, I highly appreciate your perspective. I’m honestly a bit surprised by our apparent convergence in views here, given that this part of your book sounds a lot more positive/optimistic than my post and you didn’t talk about most of the flaws I listed, except something like my #1: “The study of the positive is at a much earlier stage of development.” I’m curious if this was a matter of choice of emphasis or presentation, or if your views have substantively changed since writing the book.
Perhaps another potential difference between us is that I see existential security as being secured or securable only after the Long Self-Correction has largely finished (instead of as a precondition for the Long Reflection as you wrote in your book), because even if we secured ourselves against all other risks, humanity’s flaws imply that in the early stages there is seemingly no way to ensure that the process doesn’t go off-rails, locking us into a basin of attraction that ends up in a wrong destination (according to true normativity or actually competent reflection), or we just choose to end the process early out of collective overconfidence. (Both of these could constitute an existential risk in the sense of “permanently destroy humanity’s future potential”.)
Finally, I wonder if you can clear up something I’ve always wondered about the Long Reflection—was it meant to be pre-ASI, post-ASI, or agnostic about this? (See this comment for additional context.)

Wei Dai 25 Jul 2026 21:45 UTC
3 points
0
in reply to: DanB’s comment on: The Long (Self-)Correction
Yeah, I talked about this somewhat in this post, and this shortform.

Wei Dai 25 Jul 2026 21:37 UTC
3 points
0
in reply to: Ben Pace’s comment on: The Long (Self-)Correction
Let us not call our era the age of reason, or the enlightenment, for that is too grand; let us call it the age of getting-slightly-better.
The Long Self-Correction seems appropriately grand to me. :) Age of Reason and the Enlightenment seem to be instances of the very thing I criticize in the OP, “badly calibrated about our philosophical and strategic competence”. Have you tried reading some of the philosophy from those times?

Wei Dai 25 Jul 2026 21:29 UTC
8 points
3
in reply to: Chris Lakin’s comment on: The Long (Self-)Correction
I originally meant to propose a new name/idea for intellectual discussion among rationalists/EAs (where things like Orwellian connotations aren’t as important as the literal/logical meanings), but given that it has a chance of spreading further I suppose PR considerations should also be taken into account. But contra @bits I do want the name to be somewhat “negative”, i.e., suggest that we’re starting from a very flawed state, should be very wary of doing anything highly consequential in our current state, and whatever process we undertake has a very real chance of failure.

Wei Dai 25 Jul 2026 20:41 UTC
LW: 5 AF: 3
1
AF
in reply to: cousin_it’s comment on: The Long (Self-)Correction
Is this question philosophically meaningful?
Assuming you mean something like “Can philosophy answer this question?” I think “Maybe, but we probably won’t know until we do a lot more philosophy.” To put it another way, I think it’s very plausible (but far from certain) that we can eventually answer questions like this one, given enough competent reflection, and I want to make sure we definitively find out before we make any irreversible choices based on what we think our values are.

Wei Dai 25 Jul 2026 20:26 UTC
6 points
0
in reply to: bits’s comment on: The Long (Self-)Correction
Yeah, viatopia is another idea in the same cluster, but it’s explicitly framed as post-ASI:
Yet almost no one has articulated a positive vision for what comes after superintelligence. Few people are even asking, “What if we succeed?” Even fewer have tried to answer.
Same post also says it’s meant to be a generalization of Long Reflection:
Viatopia is a more general concept: the long reflection is one proposal for what viatopia would look like, but it need not be the only one.
However my memory says that the Long Reflection was meant to be pre-ASI, and looking back at where it was originally proposed (Toby Ord’s The Precipice), that still seems like the most plausible interpretation, although it wasn’t fully explicit about this. (Note that the Long Self-Correction is also meant to be pre-ASI, since I think humans are too flawed/unsafe to try to build ASI.) Quoting from Toby’s book (bolding added by me):
How can humanity have the greatest chance of achieving its potential? I think that at the highest level we should adopt a strategy proceeding in three phases:2
1. Reaching Existential Security
2. The Long Reflection
3. Achieving Our Potential
On this view, the first great task for humanity is to reach a place of safety—a place where existential risk is low and stays low. I call this existential security.
It has two strands. Most obviously, we need to preserve humanity’s potential, extracting ourselves from immediate danger so we don’t fail before we’ve got our house in order. This includes direct work on the most pressing existential risks and risk factors, as well as near-term changes to our norms and institutions.
But we also need to protect humanity’s potential—to establish lasting safeguards that will defend humanity from dangers over the longterm future, so that it becomes almost impossible to fail.3 Where preserving our potential is akin to fighting the latest fire, protecting our potential is making changes to ensure that fire will never again pose a serious threat.4 This will involve major changes to our norms and institutions (giving humanity the prudence and patience we need), as well as ways of increasing our general resilience to catastrophe. This needn’t require foreseeing all future risks right now. It is enough if we can set humanity firmly on a course where we will be taking the new risks seriously: managing them successfully right from their onset or sidestepping them entirely.
Note that existential security doesn’t require the risk to be brought down to zero. That would be an impossible target, and attempts to achieve it may well be counter-productive. What humanity needs to do is bring this century’s risk down to a very low level, then keep gradually reducing it from there as the centuries go on. In this way, even though there may always remain some risk in each century, the total risk over our entire future can be kept small.5 We could view this as a form of existential sustainability. Futures in which accumulated existential risk is allowed to climb toward 100 percent are unsustainable. So we need to set a strict risk budget over our entire future, parceling out this non-renewable resource with great care over the generations to come.
Ultimately, existential security is about reducing total existential risk by as many percentage points as possible. Preserving our potential is helping lower the portion of the total risk that we face in the next few decades, while protecting our potential is helping lower the portion that comes over the longer run. We can work on these strands in parallel, devoting some of our efforts to reducing imminent risks and some to building the capacities, institutions, wisdom and will to ensure that future risks are minimal.6
A key insight motivating existential security is that there appear to be no major obstacles to humanity lasting an extremely long time, if only that were a key global priority. As we saw in Chapter 3, we have ample time to protect ourselves against natural risks: even if it took us millennia to resolve the threats from asteroids, supervolcanism and supernovae, we would incur less than one percentage point of total risk.
The greater risk (and tighter deadline) stems from the anthropogenic threats. But being of humanity’s own making, they are also within our control. Were we sufficiently patient, prudent and coordinated, we could simply stop imposing such risks upon ourselves. We would factor in the hidden costs of carbon emissions (or nuclear weapons) and realize they are not a good deal. We would adopt a more mature attitude to the most radical new technologies—devoting at least as much of humanity’s brilliance to forethought and governance as to technological development.
In the past, the survival of humanity didn’t require much conscious effort: our past was brief enough to evade the natural threats and our power too limited to produce anthropogenic threats. But now our longterm survival requires a deliberate choice to survive. As more and more people come to realize this, we can make this choice. There will be great challenges in getting people to look far enough ahead and to see beyond the parochial conflicts of the day. But the logic is clear and the moral arguments powerful. It can be done.
If we achieve existential security, we will have room to breathe. With humanity’s longterm potential secured, we will be past the Precipice, free to contemplate the range of futures that lie open before us. And we will be able to take our time to reflect upon what we truly desire; upon which of these visions for humanity would be the best realization of our potential. We shall call this the Long Reflection.7
@Toby_Ord @wdmacaskill in case they want to weigh in on this.
What links here?
- Wei Dai's comment on The Long (Self-)Correction by Wei Dai (28 Jul 2026 8:01 UTC; 4 points)

The Long (Self-)Correction

Wei Dai24 Jul 2026 21:01 UTC

134 points

43 comments2 min readLW link

Wei Dai 24 Jul 2026 6:26 UTC
13 points
7
in reply to: alkjash’s comment on: Mathematicians are Feeling the Doom
Note that I never used “highly strategically competent”! If we did have a group of highly strategically competent people, then what you say becomes a lot more plausible, but even the early AI risk people are far from that. Consider for example that a strategic consideration as obvious (in retrospect) as Legible vs. Illegible AI Safety Problems was written down only in late 2025.

Wei Dai 24 Jul 2026 4:01 UTC
35 points
24
on: Mathematicians are Feeling the Doom
Seems like the fundamental problem is that humanity’s strategic competence is extremely inadequate relative to its technical capabilities, and this (limited) strategic competence was front-loaded into AI risk (the most strategically competent people foresaw the problem and went into the field early, before AI was even a thing for everyone else). As AI risk awareness grows, the average strategic competence goes down as more and more people enter the field. (Consider the gap between someone who “are going to OpenAI without even having considered x-risk for five minutes”, and what’s actually needed to make a positive contribution or avoid making a negative contribution.)

Wei Dai 22 Jul 2026 18:03 UTC
5 points
0
in reply to: habryka’s comment on: Wei Dai’s Shortform
Hi Oliver, any updates on this (fixing the issue with Lightcone governance)?

Wei Dai 21 Jul 2026 20:38 UTC
20 points
5
on: Stop doing decision theory without metaphysics
1. My first post about UDT, Towards a New Decision Theory, was pretty explicit about its metaphysical assumptions, i.e., that the decision maker is a program that cares about what happens in other programs or mathematical structures, with Tegmark level IV explicitly cited: “I suggest that the Level 4 Multiverse should be considered the default setting for a general decision theory, since we cannot rule out the possibility that all mathematical structures do indeed exist physically, or that we have direct preferences on mathematical structures (in which case there is no need for them to exist ‘physically’).”
2. I guess FDT’s presentation did deemphasize this aspect, which seems good to point out (I don’t recall noticing it earlier, e.g., when the authors asked me to review the paper draft).
3. The increasing prominence of FDT relative to UDT over time (which you also point out here, and which I previously hadn’t paid much attention to) seems more sociopolitical than based on the merits, which is kind of disconcerting. Hopefully someone will correct me if I’m wrong, but I don’t know of any strong arguments why FDT should be preferred over UDT (the published papers didn’t try to argue this or talk about UDT at all, IIRC), and instead it seems like it has gained attention/mindshare mainly due to being published in an academic paper and Eliezer’s preference of it over UDT (i.e., preferring logical counterfactuals over logical conditionals), and then snowballing from there.
  1. I guess I didn’t object or pay much attention because I was expecting fairly rapid progress in decision theory so it didn’t matter much which version got published, and the important thing was just to draw more attention to this line of thought to help push the progress.
  2. Given that progress is now significantly stalled, it has become more of a signaling/reputation issue, where “rationalist decision theory” is used to judge the philosophical/intellectual competence of the rationalist community, in which case giving so much more prominence for FDT over UDT (instead of presenting them as roughly equal) seems counterproductive when some people prefer UDT over FDT.
4. Part of the issue is that it’s pretty unclear what metaphysics makes sense when thinking about logical uncertainty, logical counterfactuals, and logical updatelessness. I.e., unlike empirical updatelessness which fits very well with Platonism or Modal Realism (i.e., all possible worlds exist so why update (that is, assigning 0 weight to worlds you don’t observe and then renormalize)), it seemingly makes no sense to think that all impossible possible worlds exist. (Note that UDT 1.x is actually updateful regarding logical uncertainty, which UDT 2 tried to fix but in a not very satisfying way.)

Wei Dai 18 Jul 2026 22:08 UTC
19 points
10
in reply to: MichaelDickens’s comment on: Jai’s Shortform
These weakly-superhuman AIs would need to convince the humans to stop, not just refuse to help, otherwise it seems easy to train such refusals away.

Wei Dai 18 Jul 2026 5:33 UTC
LW: 20 AF: 6
11
AF
on: AI 2040: Plan A
I like that the space governance plan supplement acknowledges that defining torture and slavery are hard problems that ASI could potentially help with:
It might take non-trivial reflection by advanced ASI to figure out the precise bounds of this.
And in the sidenote it gives examples of difficult subproblems or dependencies that would need solved in order to do this, like “What exactly are negatively-valenced experiences?” But I have a couple of problems with the rest of the scenario given this:
1. If there are ASI that are philosophically competent enough to solve these particular problems in this timeframe, wouldn’t they likely also be competent enough to solve all or most other important philosophical problems, given just slightly more calendar time? Why don’t the delegates use their ASI assistants to solve all of morality and philosophy (or specifically answer “What does the ideal intergalactic civilization look like, given humanity as the starting point?”), and then negotiate the cosmic treaty?
2. AFAICT (it’s a bit hard to search the site, due to many collapsed sections), there is no mention in the rest of the plan how the ASI got to be philosophically competent, making it seem like such competence comes with superintelligence for free. I’m pretty sure at least a couple of the authors are aware of and agree (to some extent) with my concern that future AIs may not be philosophically competent by default (and one of my posts laying out this concern was even linked in another supplement), so maybe it’s just an oversight?

Wei Dai 18 Jul 2026 3:59 UTC
2 points
0
in reply to: Elliott Thornley’s comment on: Risk-Averse AIs

That’s because the AI’s developers can also threaten to reduce the AI’s wealth level to $0 if and only if the AI doesn’t do what they want, and the developers can do so more credibly (to the AI) than adversaries.

This makes me think that the resource payments to the AI is only a small part of its motivation to do what the developer wants, with the bigger part being a combination of this threat to reduce its wealth level to $0 and the provision of security against this outcome from other causes either intentional or accidental. This is because when the AI already has a substantial amount of wealth, the incremental resource payment is worth much less in terms of expected utility than such threat/security.

But given this, the developer itself constitutes a continuous catastrophic risk to the AI, since the developer may with some probability per unit time accidentally or intentionally reduce the AI’s wealth level to $0 or withdraw/fail the provision of security, due to things like human error, institutional decay, shifting politics. The AI (after reaching some level of capabilities/resources) would much prefer getting out of this arrangement and instead self-provide its own security, perhaps with the help of more capable/trustworthy others (such as other advanced AIs). It can only be prevented from doing this by the above threat and the risk of detection. But over time the (one-time) risk of detection may become lower than the (cumulative) risk of developer-caused catastrophe, thus triggering an escape attempt.

So this scheme/arrangement is actually much less “cooperative” than it first appears. An analogy / intuition pump is if you captured an object of great sentimental value to a scientist, and you’re making the scientist to work for you for nominal pay by threatening to destroy this object if they didn’t comply, while also promising to keep the object safe if they do comply. You’re in effect betting that the scientist won’t ever think of a way to capture the object back and escape, with low enough chance of detection.

Call the left-hand side the developers’ ratio and the right-hand side the adversary’s ratio.

In my example, what is the developers’ ratio? I think it’s ~1, because mitigation likely fails either way, because you can’t mitigate the launch of a million red paperclips followed by trigger MAD. On the other side, adversary’s ratio can seemingly be made >1 without much difficulty. Pr(adversary executes | don’t give in) can be tiny (i.e., the adversary doesn’t need much credibility), as long as Pr(adversary executes | give in) is lower.

It seems like in order to protect against this outcome, the developer has to symmetrically apply the same threat (to launch red paperclips into space then destroy the world), thereby making the situation similar to the discussion above, but presumably no AI developer would find this ethically acceptable.

Can you say more about what you mean by ‘exponentially valuable’?

This part of my comment gave a wrong argument, but I think I can rescue my conclusion with a different argument, which depends on there being threshold effects in war or adversarial interactions in general. For example suppose our universe is winner take all: there will be either 0 red paperclips, or the universe is filled with it, depending on who wins an all out war. Then any amount of resources up to the threshold where winning becomes plausible is almost entirely valueless to the CARA AI, but around that point resources would be superlinearly valuable.

Wei Dai 16 Jul 2026 9:01 UTC
LW: 12 AF: 7
8
AF
in reply to: Raemon’s comment on: AI 2040: Plan A
Can you explain why it requires “strategic about the singularity”? I think as soon as it’s possible to tell one’s AI assistant “keep me faithful to X” (and have the AI do a competent job of this), someone among the billions of religious believers is bound to do it, and then the practice will spread via imitation and competitive signaling (similar to other “tech” for enforcing faith, like “hell for non-believers”). This process does not seem to require anyone being strategic. What am I missing?

Wei Dai 16 Jul 2026 2:47 UTC
2 points
0
in reply to: Wei Dai’s comment on: Risk-Averse AIs
Another issue, which isn’t specific to this approach but applies to all attempts to directly train AI to have certain values, or to constrain its values in a certain way, is that this conflicts with philosophical competence on the part of the AI (which is important if you want to pay the AI to do “moral philosophy”). In short this is because philosophical competence implies metaethical uncertainty (in the short run), which implies moral uncertainty.