Wei Dai

Karma: 47,406

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 9 Aug 2026 17:31 UTC
21 points
2
on: What just happened? A retrospective of AI alignment
I feel like this is giving way too little weight/salience to humans being really bad at strategy and philosophy in general, and in particular MIRI being bad at strategy and philosophy. You talk about Holden being misled about AGI for 8 years, but don’t mention MIRI planning to build recursively improving Friendly AI with a small team and potentially just 1 philosopher, for a comparable amount of time. If OpenPhil had funded MIRI more in 2016, it would have been funding them to attempt this!
Just in course of searching for my name in the comments section of Holden’s Thoughts on the Singularity Institute (SI), I came across more examples (of humans being bad at strategy and philosophy):
1. Eliezer banning Roko’s Basilisk post (and Roko posting it in the first place)
2. Eliezer apparently forgetting or disregarding most previous critics of MIRI (e.g. me): “Nonetheless, it already has a warm place in my heart next to the debate with Robin Hanson as the second attempt to mount informed criticism of SIAI.”
3. Eliezer forgetting about banning Roko’s post, writing in all caps “Once again: ROKO DELETED HIS OWN POST. NO OUTSIDE CENSORSHIP WAS INVOLVED.”, and komponisto chickening out of reminding Eliezer about it.
4. cousin_it avoiding the debate because “I realized that talking about saving the world makes me really upset and I’m better off avoiding the whole topic”
5. MIRI hiring Ben Goertzel as Research Director even though Eliezer thought his AGI project would kill everyone if it succeeded (“And if Novamente should ever cross the finish line, we all die.”), because “Ben Goertzel’s projects are knowably hopeless, so I didn’t too strongly oppose Tyler Emerson’s project from within SIAI’s then-Board of Directors; it was being argued to have political benefits, and I saw no noticeable x-risk so I didn’t expend my own political capital to veto it, just sighed. Nowadays the Board would not vote for this.”

Wei Dai 9 Aug 2026 11:01 UTC
13 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I noticed @Eliezer Yudkowsky distancing himself from Situational Awareness LP, without mentioning the connection via @CarlShulman who used to be a MIRI employee, and also that SALP just made a $500 million investment in a chip manufacturing startup, suggesting that Carl seems to be even more comfortable with accelerating AI development these days. (Unlike SALP’s investments in Anthropic and various public companies, which can be somewhat plausibly explained as safety motivated or not counterfactually meaningful, it seems hard to explain away the acceleration effect of investing in a pure hardware startup.)

Wei Dai 8 Aug 2026 12:30 UTC
6 points
0
on: When does competition lead to recognisable values?
Do you personally prefer to live in a world with a lot of slack (like today’s world) or in a Malthusian world where you’re barely above subsistence? The former, right? So even if Malthusian competition among AIs can recreate human-like values, that world would be dispreferable (have low value) according to those values!
In the original blog post, we think a lot about slack. It says that if you have slack, you can kind of go off the optimal solution and do whatever you want. But in practice, what we see is that slack, when it occurs, produces this kind of drift. It’s basically the universe fulfilling its naturally entropic nature, in that most ways to go away from the optimum are bad. If we randomly drift, we just basically tend to lose fitness and produce really strange things which are not even really what we value.
This random drift seems obviously caused by humans’ lack of sufficient intelligence/agency/cooperation, not some sort of fundamental problem with slack? Also, where is “the original blog post” referred to here?

Wei Dai 8 Aug 2026 11:50 UTC
11 points
−7
on: FAQ: Isn’t AGI coming too soon for reprogenetics to help?
HIA may indirectly slow down AGI capabilities
Another potential route for this is that reprogenetics can help solve the fertility crisis, which can reduce the incentives to build AGI. My thinking here is that currently it’s pretty risky to have children (e.g. if even one of them ends up unhealthy or “unsuccessful” / low status, it’s a big drain on one’s own finances/status/happiness) which I think contributes to low desire to have more kids in a lot of societies, and low average fertility is a motivation for building AGI in order to stop/reverse the resulting economic or civilizational decline. Reprogenetics could help fix this root cause.
Although I suppose this depends on reprogenetics being cheap enough. If it’s too expensive (and can’t be or isn’t subsidized), it might have the opposite effect, where people feel like it’s a de facto requirement of parenthood (e.g. they feel really bad about their unenhanced kids having to compete with enhanced kids) and opt out of having children because they can’t afford it.

Wei Dai 8 Aug 2026 4:47 UTC
3 points
0
in reply to: Rauno Arike’s comment on: Rauno’s Shortform
I second this, and also wish for a pingback/backlinks feature for comments (both regular display and notifications). GreaterWrong displays backlinks to and from comments, which I’ve been using regularly, but the site has been down most of today and given that Said has been banned I don’t know how much motivation he has to maintain the site. Unfortunately LW’s API doesn’t support this, and GW does it by fully cloning the LW database and then scanning for backlinks server-side, so I can’t just implement this in my userscript. @habryka @Raemon

Wei Dai 8 Aug 2026 1:32 UTC
21 points
1
on: Wei Dai’s Shortform
Trying to explain more of my thinking style, following up to Counterintuitive Comparative Advantage and Trying to understand my own cognitive edge. I recall distinct childhood memories of:
1. disliking “unverifiable” classes/topics like creative writing and analyzing literature due to lack of straightforward right or wrong answers
2. scared of the possibility of making a highly consequential mistake, like killing a patient as a doctor
3. anxious at the prospect of having to produce insights on a deadline, like I imagined a researcher would
Which together almost fully explains my initial major/career of CS/programming, but makes it super ironic that I now “do research” in philosophy and macrostrategy. Some of my childhood preferences seem to show through however:
1. I seem more aware than most philosophers/strategists that I work in “unverifiable” domains so there could easily be mistakes or good counterarguments that I’m missing, therefore I should be very wary of making any strong claims or having high confidence in my conclusions.
2. I generally only talk about abstract ideas, which feels safer in that if someone agrees with me and translates them into concrete actions, which turn out to be net negative, I can’t be blamed too much. Even then I hesitate to push my ideas too hard (like should I remind people of Legible vs. Illegible AI Safety Problems in light of certain AI safety problems suddenly becoming very legible in recent days).
3. I’ve avoided joining or affiliating with any orgs, partly in order not to have any pressure to be productive.

Wei Dai 7 Aug 2026 20:50 UTC
2 points
11
in reply to: Wei Dai’s comment on: Arguments for P
I wonder what the agreement voters are agreeing with here… Also, I wonder what the * * * horizontal break in the OP means. What’s separates the first and second group?

Wei Dai 6 Aug 2026 10:13 UTC
47 points
4
in reply to: Kabir Kumar’s comment on: Arguments for P
It vividly illustrates something I’ve been wondering about for a while, namely why even the closest humans (e.g. Paul and I, who share very similar backgrounds and interests, even independently inventing the same ideas) are so different in thinking styles and conclusions.
What links here?
- Wei Dai's comment on Wei Dai’s Shortform by Wei Dai (8 Aug 2026 1:32 UTC; 21 points)

Wei Dai 6 Aug 2026 10:05 UTC
67 points
15
on: Arguments for P
Some people claim to be P anti-realists (which BTW is yet another instance of philosophical/strategic overconfidence), but it seems impossible to be a meta-P anti-realist. Unfortunately AIs are even worse at reasoning about P. I don’t know what to do in light of all this, but will keep thinking about it for the benefit of our potential simulators.

Wei Dai 4 Aug 2026 8:13 UTC
LW: 5 AF: 5
0
AF
on: Notes on technical alignment via human-like social drives
I don’t think that theory [social status] explains quite as much as those people think it does
Do you have a link/explanation for this? I think this may be fairly cruxy, because I’m guessing your intuitions for truth-seeking disagreeable nerd AGI are substantially based on truth-seeking disagreeable nerd humans, so it matters what those humans’ real motivations are.
One line of thought I have here is, there are lots of things such a human or AGI could disagree or talk about or have an interest in, how does it pick which one? I think for the human it probably comes down to some kind of subconscious status calculation, but in either case, how does the AGI do it if it doesn’t have its own status motivations or other long-term goals?

Wei Dai 4 Aug 2026 7:35 UTC
3 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I posted a screenshot of this shortform to X on a whim^[1] and it ended up getting 238k views, 1.1k likes, and increased my follower count from ~9k to ~13k. One observation is unlike when I started tweeting ~3 years ago, there was no pushback from e/acc, which seems to have completely disappeared from X.
1. ^
  thought I could maybe catch some algorithmic coattails from the “situational awareness” situation

Wei Dai 3 Aug 2026 23:16 UTC
LW: 5 AF: 4
0
AF
in reply to: Max Harms’s comment on: A Conflict Between AI Alignment and Philosophical Competence
I don’t think there’s any tension for the frames of corrigibility that I prefer, where the corrigible agent terminally-values having a certain kind of relationship with the principal.
A philosophically competent agent would seemingly (until it fully solved morality/axiology) have uncertainty about any terminal values, including “having a certain kind of relationship with the principal”. So I don’t understand how your approach or framing gets away from my critique in the OP. (Would be interested in having a chat about this if that might help.)

Wei Dai 31 Jul 2026 9:12 UTC
3 points
0
in reply to: Ondřej_Kubů’s comment on: The Long (Self-)Correction
Thanks for the pointer, which I didn’t know about, but yeah I mostly mean discussing/analyzing the relationship between status seeking and altruism, strategizing how to best take advantage of it, including how to avoid problems caused by misalignment between the two (like in this example), and not just going ahead and pulling the lever as you put it.

Wei Dai 29 Jul 2026 19:25 UTC
6 points
0
in reply to: habryka’s comment on: Some Dating Stories
That was my memory/assessment, but of course I’d be biased in that direction. I guess the only evidence I can easily point out (for a large role in things dying down) is that Alicorn was tracking all of the top-level posts on the topic, and nothing came after mine. I’d be interested in anyone else’s take, if they were around at the time and remember this drama. I guess @Roko should remember, since it he was centrally involved (and wrote the comment that triggered the main debate).
What about the other posts that Alicorn tracked? Do you think the old debate was reasonably complete, or have additional thoughts/arguments that none of them brought up?

Wei Dai 29 Jul 2026 10:48 UTC
5 points
0
on: You can, in fact, bamboozle an unaligned AI into sparing your life
Are we in a simulation? What should we do?
I missed your call for links earlier. I’ve written a couple of pieces about this:
- Beyond Astronomical Waste (in short, try harder to ensure philosophical competence of future AIs or humans or civilization as a whole)
- https://www.lesswrong.com/posts/HbkNAyAoa4gCnuzwa/wei-dai-s-shortform?commentId=NrtKECeYASwoBJbki (“thinking and writing with potential simulators in mind as an additional audience, hoping my ideas might benefit or interest some of them even if they end up largely ignored in this reality”)

Wei Dai 28 Jul 2026 19:00 UTC
5 points
0
in reply to: habryka’s comment on: Some Dating Stories
Have you seen The Nature of Offense? I remember feeling very proud of this post and thinking that it kind of “saved” LW from the PUA war at the time. (It came last in a sequence of posts debating the controversy by many prominent posters, and got the highest karma by a large margin. Afterwards, the debate mostly died down, and people seemed to stop posting PUA content voluntarily.) @Ben Pace

Wei Dai 28 Jul 2026 8:31 UTC
3 points
0
in reply to: WillPetillo’s comment on: The Long (Self-)Correction
An acknowledged scope limitation is not the same as a flaw, and I’m not sure which of these you mean by “problem.” The explicit purpose of an AI Pause is to create time for some intentionally unspecified longer-term solution, on the grounds that there are many competing theories as to what such a solution looks like, whereas needing time to implement—and decide between them—is a common factor.
Good point/question. My intended meaning is closer to the former, and basically the “problem” is referring to the fact that I lacked a good handle for the concept that I often want to invoke, with “AI Pause” and “Long Reflection” being the closest ones. I think what you say here makes sense and I don’t intend for my new handle to replace “AI Pause” where “AI Pause” is more appropriate.
The rest of your comment seems like an interesting idea for solving part of the challenge, worth keeping in mind, and fleshing out and discussing in the future, when we have this “generalized externalized cost tax” that we can build your idea into.

Wei Dai 28 Jul 2026 8:01 UTC
13 points
1
in reply to: Toby_Ord’s comment on: The Long (Self-)Correction
Thanks, Toby, I highly appreciate your perspective. I’m honestly a bit surprised by our apparent convergence in views here, given that this part of your book sounds a lot more positive/optimistic than my post and you didn’t talk about most of the flaws I listed, except something like my #1: “The study of the positive is at a much earlier stage of development.” I’m curious if this was a matter of choice of emphasis or presentation, or if your views have substantively changed since writing the book.
Perhaps another potential difference between us is that I see existential security as being secured or securable only after the Long Self-Correction has largely finished (instead of as a precondition for the Long Reflection as you wrote in your book), because even if we secured ourselves against all other risks, humanity’s flaws imply that in the early stages there is seemingly no way to ensure that the process doesn’t go off-rails, locking us into a basin of attraction that ends up in a wrong destination (according to true normativity or actually competent reflection), or we just choose to end the process early out of collective overconfidence. (Both of these could constitute an existential risk in the sense of “permanently destroy humanity’s future potential”.)
Finally, I wonder if you can clear up something I’ve always wondered about the Long Reflection—was it meant to be pre-ASI, post-ASI, or agnostic about this? (See this comment for additional context.)

Wei Dai 25 Jul 2026 21:45 UTC
3 points
0
in reply to: DanB’s comment on: The Long (Self-)Correction
Yeah, I talked about this somewhat in this post, and this shortform.

Wei Dai 25 Jul 2026 21:37 UTC
3 points
0
in reply to: Ben Pace’s comment on: The Long (Self-)Correction
Let us not call our era the age of reason, or the enlightenment, for that is too grand; let us call it the age of getting-slightly-better.
The Long Self-Correction seems appropriately grand to me. :) Age of Reason and the Enlightenment seem to be instances of the very thing I criticize in the OP, “badly calibrated about our philosophical and strategic competence”. Have you tried reading some of the philosophy from those times?