Wei Dai

Karma: 48,448

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 28 Sep 2026 0:48 UTC
10 points
0
on: Wei Dai’s Shortform
(Cross-posted from X)

Why didn’t more AI safety people advocate for AI pause/stop earlier? Well, aside from sociological reasons, in order to do that, you had to think that none of the following would work out (be feasible, safe, have low enough safety tax) in time, which takes a degree of skepticism almost no one could muster. (Or object to AGI/ASI on non-consequentialist grounds, but there was apparently a very high correlation between consequentialism and early interest in AI safety.)

Friendly AI (CFAI)
Coherent Extrapolated Volition (CEV)
Metaphilosophical AI
Tool AGI
Value Learning
Oracle AI
Agent Foundations
Corrigible AI
Quantilizers
Approval-Directed Agents
Human Imitation
Cooperative Inverse RL (CIRL)
Iterated Distillation and Amplification (IDA)
Task-Directed AGI
RL from Human Feedback (RLHF)
Impact Regularization
Mechanistic Interpretability
AI Safety via Debate
Recursive Reward Modeling
Comprehensive AI Services (CAIS)
Infra-Bayesianism
Alignment by Default
Natural Abstraction
Eliciting Latent Knowledge (ELK)
Shard Theory
Constitutional AI
AI Control
Weak-to-Strong Generalization

To recenter the sociology, it was much easier to build a career out of being bullish one or more of these approaches, than out of general skepticism. (I’ve been independent, financially and otherwise, throughout my participation in AI safety, which was perhaps not a coincidence from being the only AI pause/stop advocate for a long time.)

Wei Dai 25 Sep 2026 0:13 UTC
8 points
1
in reply to: Cleo Nardo’s comment on: Overtly misaligned trajectories score highly in RL.
>I’d be surprised if they were using anything much cleverer than a cheap monitor checking for egregious reward hacks using a rubric.
This is also my impression, but what I’m not sure about is whether this is the result of them having tried pretty hard to do scalable oversight, and that effort failing (and therefore not reaching production), or not trying very hard so far. (I guess OpenAI probably hasn’t tried very hard, with the Superalignment team leaving and all, but what about Anthropic?)

Wei Dai 24 Sep 2026 23:50 UTC
21 points
0
on: Overtly misaligned trajectories score highly in RL.
Scalable oversight. Maybe we could build better harnesses of the LLM graders (e.g. debate with cross examination plus plus) so they match human performance.
I wonder how hard the AI companies have already tried on this (hence how much room there is for improvement in the short term). It seems hard to get a good sense of this from the outside. Does anyone have an insider perspective (or even rumors about this) they can share?

Wei Dai 14 Sep 2026 14:52 UTC
10 points
8
in reply to: habryka’s comment on: Personal statement on joining the OpenAI nonprofit board
I wonder if you perhaps missed that I was responding to these tweets from Eliezer, which already made the tribal distinction (that I was in part trying to push back against)? (Otherwise I’m having trouble understanding why you’re annoyed at me in this conversation.) Edit: I guess I also had some genuine curiosity as to what Eliezer and others saw as the differences between EA and rationalists, so some parts of the thread was more about that than pushing back. Still hard to see anything that I did as wrong or annoying… Also, totally fine not to engage, just wanted to express my confusion.

Wei Dai 11 Sep 2026 22:56 UTC
4 points
1
in reply to: TsviBT’s comment on: PauseAI Has ‘officially disendorsed’ PauseAI-US
Your notion of “morally bad actor” does not seem to require/imply that the person “know what they’re doing is bad”, because you could convert someone who doesn’t know this into being a friend, by convincing them that what they’re doing is bad, either through arguments or by applying social pressure. This seems to be what Liron and Holly are in part trying to do?
(I can also imagine Liron/Holly using the term in a different way, which also seems reasonable, more in line with my example of a torturer AI, which is that this agent is doing something really bad according to our morality, and so we should mobilize/coordinate to do something about it.)

Wei Dai 11 Sep 2026 17:57 UTC
3 points
0
in reply to: TsviBT’s comment on: PauseAI Has ‘officially disendorsed’ PauseAI-US

I think this doesn’t actually follow. It does matter whether they know what they’re doing is bad. If they know and continue, they’re a morally bad actor.

(Assuming you mean this as a biconditional “if and only if”.) By this standard, if a future AI decides to torture a bunch of humans in the course of taking over the world (and this isn’t bad according to their values/decision theory/philosophy), they’re not a morally bad actor. Do you endorse this?

Wei Dai 11 Sep 2026 1:47 UTC
24 points
6
in reply to: David Matolcsi’s comment on: Personal statement on joining the OpenAI nonprofit board
In general, it doesn’t seem that bad policy to me that some thoughtful people should try to specialize in joining powerful institutions and talk sense to them at the cost of their public voice, while others should remain fully independent and try to become honest and unbiased public thought-leaders.
Isn’t it crazy that our world makes these choices mutually exclusive, and on a meta level, everyone just takes it in stride?
BTW who are some remaining public thought-leaders with views similar to Paul’s? I often wonder “what does Paul think about this development or idea?” and I’m not sure whose posts/comments to look up for the closest substitute. I guess Geoffrey Irving comes to mind but he doesn’t post/comment nearly as much as Paul did.
Here’s a list made by Perplexity^[1], but none really fit a combo of prolific poster/commenter, independent voice, and focus on scalable pragmatic alignment, that Paul represented before joining USG:
- Rohin Shah for the most regular technical commentary and best coverage of the whole practical-alignment portfolio.
- Jan Leike for the closest direct continuation of scalable oversight and automated alignment.
- Geoffrey Irving for debate and formal scalable-oversight protocols.
- Evan Hubinger for deception/inner-alignment pressure-testing.
- Buck Shlegeris for control, monitoring, and the “what if the model is actively trying to bypass oversight?” perspective.
- Beth Barnes / METR for empirical evaluations and evidence standards.
1. ^
  prompt: who are some people with views similiar to Paul Christiano, especially on the need for scalable alignment? who among them posts/comments the most?

Wei Dai 27 Aug 2026 22:57 UTC
21 points
0
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
Some points (cleaned up by LLM) I made during an out-of-band chat with @Raemon (all of which, IIUC, either didn’t update him much or are not cruxy for him):^[1]
Criticism is rare For example, UDT didn’t get much criticism — see UDT shows that decision theory is more puzzling than ever, where I had to find a lot of the issues with it myself. (Daniel wrote up the commitment race problem, but I had found and talked about it previously.) So, like some other LW commenters, I don’t think hobby-horsing is bad: someone making even a not-that-central critique is at least better than nothing, if it corrects some error in the post — like I think my comment to Richard did. (Reviews don’t fix this: there are very few of them compared to comments, and writing an overall assessment is a different skill from bug hunting — I’ve never managed to write a single annual review myself.)
Critics can’t reliably tell which of their critiques are “central.” Part of my implicit model: critics typically won’t share the same understanding of an author’s idea as the author himself — either they haven’t put in as much effort to understand it, or the idea is actually nonsense but the author is too biased to see it. So their sense of what critiques are central is likely to be pretty different from the author’s. Asking critics to avoid non-central critiques, explicitly or implicitly via the threat of deletion, makes people less likely to make critiques at all, because they don’t want to risk doing something considered wrong / low-status according to local culture (or just waste their effort, or not have fun). The only solution I can see is to encourage people to make all the critiques they can think of, and make the author resilient enough to just ignore the non-central ones, or say something like: “Really appreciate your effort in making a critique. Unfortunately, according to my understanding of my idea (which I do not expect you or others to fully share), I don’t think this is central, so I’m not going to address it, at least for now. Feel free to elaborate more if you do strongly suspect it is central, or I may come back to it myself in the future.”
I’ve had this concern since 2010. See Tips and Tricks for Answering Hard Questions: “Don’t stop at the first good answer.” That was the LW1 era. LW2 seems to be going in the opposite direction (not exactly, but at least in an orthogonal direction that takes us further from the destination I envision).
To clarify my slippery slope concern. I’ve been told the target is “authors feel free to moderate their posts the way they would their Facebook wall, but people are encouraged to write up responses elsewhere and those responses show up in Mentioned In.” To clarify: that is the end of the slippery slope I’m worried about, not something I’m worried about slipping past. I do also worry it could slip a bit further, in the sense of people finding new reasons to ban people — like the reason cited for banning Said, and the hobby horse idea — and being more willing to apply existing such ideas, so banning becomes more frequent (or people become more discouraged to criticize even if the frequency of banning stays the same), even as the hard rules stay constant and site mods don’t try to push the culture more.
A concrete compromise proposal. A modified version of the compromise from this comment: create a new type of content similar to shortform but in a different place (so they don’t clutter up shortforms) — call them “off-comments” for now. Have “Mentioned In” include these as links, with the “Mentioned In” section fully expanded by default. There’s also a collapsed-by-default section at the bottom that inlines all of these off-comments (and their reply threads); when you click a link in “Mentioned In” it expands that collapsed section and navigates to the inlined off-comment. Authors can’t ban people from making off-comments (at least without site mod approval), and when they delete a comment the commenter has a button/option to convert that comment into an off-comment.
1. ^
  Writing them down for public record, not inviting or expecting more engagement from the mods.

Wei Dai 25 Aug 2026 8:59 UTC

16 points

in reply to: leogao’s comment on: What just happened? Pragmatism and Pessimization

i don’t really think of situational awareness as “ai safety people”.

Carl Shulman is its Research Director (EDIT: and co-portfolio manager), and used to be a MIRI employee (2010-2013).

prompt: carl shulman’s publications on AI safety

Carl Shulman’s AI-safety work is concentrated in early foundational writing on superintelligent-agent alignment, instrumental convergence, intelligence-explosion dynamics, and governance rather than contemporary empirical alignment research. His publication record also includes adjacent work on digital minds, forecasting, and long-run governance.[semanticscholar][alignmentforum]

Core AI-safety publications

Publication	Year	Coauthors	Main contribution
Machine Ethics and Superintelligence	2009	Henrik Jonsson, Nick Tarleton	Argues that ordinary machine-ethics approaches make assumptions—incremental deployment, human-comparable capabilities, human institutional embedding—that may fail for agents at or beyond human level. It calls for advance analysis of the harder alignment problem. [intelligence]
Arms Control and Intelligence Explosions	2009	Stuart Armstrong	An early AI-governance paper: analyzes incentives for unsafe competitive development, winner-take-all dynamics, and the possibility that advanced AI capabilities could also make verification and international agreements more feasible. [intelligence]
Which Consequentialism? Machine Ethics and Moral Divergence	2009	Henrik Jonsson, Nick Tarleton	Discusses divergence among consequentialist objectives and its implications for constructing moral machines—relevant to the problem of specifying AI goals. It appears in his scholarly bibliography, though it is less directly focused on catastrophic-risk mechanisms than the three items above. [semanticscholar]
Omohundro’s “Basic AI Drives” and Catastrophic Risks	2010	—	Critically examines the argument that a broad class of goal-directed systems could develop instrumental tendencies such as self-preservation and resource acquisition, and relates those tendencies to catastrophic AI-risk scenarios. [intelligence]
Implications of a Software-Limited Singularity	2010/2013 version	Anders Sandberg	Explores implications of a rapid AI transition constrained primarily by software development rather than hardware or physical replication. It is important background for takeoff-speed and strategic-risk debates. [semanticscholar]
How Hard Is Artificial Intelligence? Evolutionary Arguments and Selection Effects	2012	Nick Bostrom	Addresses evolutionary and selection-effect arguments about the difficulty of developing AI, bearing on forecasts and the plausibility/timing of transformative AI. [semanticscholar]
Racing to the Precipice: A Model of Artificial Intelligence Development	2016	Stuart Armstrong, Nick Bostrom	Models competitive AI development in which actors trade safety for speed, formalizing “race to the bottom” concerns that remain central to AI governance. [dl.acm][semanticscholar]

Especially worth reading

If your interest is specifically AI x-risk and alignment, I’d prioritize:

“Omohundro’s ‘Basic AI Drives’ and Catastrophic Risks” — best for instrumental convergence and the case that misaligned optimization can be dangerous even without malicious intent.[intelligence]
“Machine Ethics and Superintelligence” — best for the conceptual distinction between conventional AI ethics and alignment for highly capable autonomous agents.[intelligence]
“Racing to the Precipice” — best for strategic dynamics: why safety measures may be underprovided under competition.[dl.acm]
“Arms Control and Intelligence Explosions” — best for governance implications and the argument that AI may create both unusual destabilization and unusual verification/enforcement opportunities.[intelligence]

Related, but broader than safety

Two later works with Nick Bostrom—Propositions Concerning Digital Minds and Society and Sharing the World with Digital Minds—are principally about moral status, rights, and social organization involving digital minds. They matter for post-AGI governance and AI welfare, but are not technical alignment papers.[80000hours][experiencemachines.substack]

For a more complete bibliography that includes papers, posts, and conversations, Shulman’s LessWrong publication timeline says formal work can be isolated by filtering its Format field to “Paper.”[lesswrong]

What links here?

Richard_Ngo's comment on What just happened? Pragmatism and Pessimization by Richard_Ngo (3 Sep 2026 17:41 UTC; 8 points)

Wei Dai 25 Aug 2026 8:40 UTC
9 points
5
in reply to: Wei Dai’s comment on: Wei Dai’s Shortform
I think most attempts to make intellectual progress are likely to fail
I wonder if one issue with people’s models of intellectual progress is that ~nobody on LW saw the ~10 years I spent exploring various dead ends / unclear results in anthropic reasoning and decision theory^[1], they just saw UDT 1.0 pop almost out of nowhere. (Perhaps some saw the few months of decision theory discussion on LW before my post, where I got some of the final pieces of the puzzle from Eliezer and Nesov’s hints.)
As for Bitcoin, I don’t know how long it took Satoshi, but there were hundreds of published papers on digital cash designs of all kinds^[2], before Bitcoin came on the scene and took off, but ~everyone just saw Bitcoin and don’t know the rest.
Sorry if this is being unfair/uncharitable to you^[3], @Richard_Ngo, but I suspect this might be what’s accounting for our apparent disagreements around intellectual progress.
1. ^
  I posted half-baked ideas to my own everything-list mailing list, but didn’t get much attention from others, except Hal Finney who liked and wrote up UDASSA on his website, which somehow spread from there even though I wasn’t all that happy with it.
2. ^
  My boss during my internship at Microsoft Research asked me to work on his own design, but I let it quietly fall off my plate because it seemed unpromising, and I had my own ideas at that point.
3. ^
  Maybe you’ve observed similar phenomenon elsewhere and still reached different conclusions than me, or I’m completely misunderstanding where our disagreement lies.

Wei Dai 24 Aug 2026 19:46 UTC
4 points
0
in reply to: Ben Pace’s comment on: Wei Dai’s Shortform
from https://intelligence.org/2021/11/11/discussion-with-eliezer-yudkowsky-on-agi-interventions/ (this is the closet thing Perplexity found)
Anonymous
Does pushing for a lot of public fear about this kind of research, that makes all projects hard, seem hopeless?
Eliezer Yudkowsky
What does it buy us? 3 months of delay at the cost of a tremendous amount of goodwill? 2 years of delay? What’s that delay for, if we all die at the end? Even if we then got a technical miracle, would it end up impossible to run a project that could make use of an alignment miracle, because everybody was afraid of that project? Wouldn’t that fear tend to be channeled into “ah, yes, it must be a government project, they’re the good guys” and then the government is much more hopeless and much harder to improve upon than Deepmind?

Wei Dai 24 Aug 2026 9:37 UTC
5 points
0
in reply to: Richard_Ngo’s comment on: Wei Dai’s Shortform
There’s also something that I (and, I infer, Habryka) want to protect—as implicitly expressed in my post it was something like “intellectual progress is in fact a valuable thing which we can aim for”. Insofar as we’re in a conflict frame with each other, you could view Wei’s original comment as an attack on that thing.
This confuses me. Can you explain what you sense as a possible motivation on my part to attack “intellectual progress is in fact a valuable thing which we can aim for”?
My own interpretation of what the conflict is, is that everyone here wants to support intellectual progress, but have different ideas how to go about it due to being biased due to status seeking, sunken costs, etc., some of which is strong/ingrained enough to make it intractable to fix the underlying mistakes, so we can only fight it out as a conflict. (To be clear I’m not certain about this.)
On your end, it would ideally involve acknowledging the thing that Habyrka and I are trying to protect, and helping us figure out how to protect it with as few tradeoffs as possible.
Does this still make sense given what I wrote above?

Wei Dai 24 Aug 2026 8:57 UTC
14 points
0
in reply to: Richard_Ngo’s comment on: Wei Dai’s Shortform
Do you have an example of intellectual progress that fits this model? “growing a new ontology/developing a new model of the world” makes sense to me, but I guess the way I’ve been doing it is to develop it mostly in private (maybe sharing it as I go but, from personal experience, I don’t expect anyone to have an interest in it, unless they independently happen to have the same intuitions / research tastes as me), until I develop some kind of minimum viable product (e.g. UDT 1.0, or the Bitcoin whitepaper as an example from someone else) based on the new ontology/model that is legibly good (e.g., clearly solves some open problems, seems clearly insightful to other people without them having to do a lot of intellectual/interpretive work, etc.).
To share more of my model, I think most attempts to make intellectual progress are likely to fail, e.g. are not aimed in a right direction. So you may think you’re growing a new ontology/developing a new model of the world, but most likely you’re just developing a bunch of nonsense but you can’t see that yet (e.g. due to lack of compute to find a contradiction in your ideas). So then if two such people try “build the bridge from both ends” that’s just very likely to not result in something useful. (I.e., it requires the two people to both be on the right track, which is multiplicatively unlikely.)

Wei Dai 24 Aug 2026 7:57 UTC
15 points
0
in reply to: Max H’s comment on: Wei Dai’s Shortform
Does my track record of being able to consistently spot important trends early, and putting serious effort into relevant work, while others thought it was too far in the future to worry about or unlikely to ever happen, sway you at all (and how much does it sway you if so)? Would really appreciate an answer even if it’s “no”, since I really want to know how useful it is to cite this in an argument like this one.
(Also have some more object-level arguments I could make, about why I’m concerned, but didn’t earlier due to laziness / allocation of effort/attention elsewhere. Let me know if you really care about hearing them.)

Wei Dai 23 Aug 2026 18:45 UTC
1 point
0
in reply to: Raemon’s comment on: Wei Dai’s Shortform
Oh I see. To explain why I “ignored” it, I thought what you wrote wasn’t directly relevant to what I was trying to do at the time, which was to test whether Lightcone might be amenable to my offer. I had my reasons for thinking that the offer would be a win for me despite what you wrote, basically because I think you didn’t quite understand how I thought it would solve (or at least possibly solve) the problem, but I didn’t feel like taking the time to explain. Because if Lightcone wouldn’t be amenable to the offer, then explaining my model of how it would help me would be kind of pointless.

Wei Dai 23 Aug 2026 18:16 UTC
3 points
2
in reply to: Raemon’s comment on: Wei Dai’s Shortform
In this case I actually did feed the thread to Gemini 3.1 Pro and asked it how habryka would answer my question, and it said 1 or 4, and then habryka’s actual answer turned out to be closer to 2. Its reasoning, which I thought was plausible, was that habryka has a vision of the archipelago that he wants to achieve, that depends on authors being mods, so the cost of site mods wasn’t really a crux for him. So I was kind of surprised when I saw habryka’s actual answer.
I suspect you may be overestimating how clear your writings/answers are, or how much you explained things, or how relevant previous explanations are given new assumptions, or how much I can/should trust you/others to represent habrya. For example, I asked earlier “Why can’t one of the other experienced mods train or manage the new mod?” and got no answer to this (confirmed via LLM that it wasn’t answered or explained anywhere in the entire comment tree rooted at the shortform OP), so there’s a big blank spot in my model of what’s going on at Lightcone.

Wei Dai 23 Aug 2026 12:19 UTC
6 points
0
in reply to: the gears to ascension’s comment on: Habryka’s Shortform Feed
My first “homes”, Usenet newsgroups and the Cypherpunks mailing list, were completely unmoderated. Anything went, including totally off topic posts, personal insults, giant flame wars, etc. A lot of that was probably not helpful for my intellectual development (or at least it’s hard to see how), but I suspect being exposed to all sorts of attacks and criticism, and not being able to deal with them except by counterargument or ignoring them, must have played an important role. (Subsequent places probably had more moderation, but mostly still little to none according to my memory.)
There’s simply no way to fully replicate that today, nor would I want to if I could, just for my own QOL, and a lot of it probably is harmful on net. I think having no direct author mod powers (only site modes, or author mods have to get site mod approval for their actions) at least replicates the plausibly most important aspect of that environment, in that it makes you often exposed to critiques that you can’t just make disappear, no matter how much they annoy or frustrate you.
Maybe there are other aspects that are important, but don’t come to my mind for now. I’ll think about this some more.

Wei Dai 23 Aug 2026 10:32 UTC
3 points
0
in reply to: Vladimir_Nesov’s comment on: Habryka’s Shortform Feed
This all seems reasonable, but what does it imply in terms of actions? What would you do if you were me or Habryka?

Wei Dai 23 Aug 2026 10:26 UTC
5 points
0
in reply to: cousin_it’s comment on: Habryka’s Shortform Feed
Reflecting on this, I think a large part of the problem is that I don’t know how to articulate a legible and convincing case against someone with a different prior. My own beliefs are probably largely based on my own experience of “growing up” in a series of intellectual environment with little to no moderation, and certainly no author moderation, and thinking that it must have contributed to my intellectual style or habits in some important way, but how I can possibly trace down the mechanisms of this or otherwise make this legible? (Plus the basic arguments that I keep repeating but don’t seem convincing to others, like criticism is good, authors are too biased, etc.)

So I’m afraid that if I give up and let the last place on the Internet that still looks a bit like my youth disappear, there will never be another Wei Dai. What would you do if you were me?
What links here?
- Habryka’s Shortform Feed by habryka (27 Apr 2019 19:25 UTC; 56 points)

Wei Dai 23 Aug 2026 8:21 UTC
2 points
−2
in reply to: habryka’s comment on: Habryka’s Shortform Feed
Sorry, LessWrong is not a charity shelter. You can make some kind of argument that people with this preference are super valuable, but this feels like a weak argument.
Invert this though, and you’re saying that the person who:
- went into x-risk and d/acc in 1995, long before these words/concepts/fields existed, by writing an open source cryptography library to help secure the net against future rogue AIs
- described an early form of cryptocurrency in 1998 and then put it aside to pursue more interesting ideas
- started arguing for not developing AI in 2004 (on SL4)
- started thinking about decision theory for future minds in early 2000s, synthesized various hints/insights into UDT in 2009, then grew skeptical of it after finding many further unsolved problems
- repeatedly argued against MIRI’s AGI/ASI plans, including in 2013, when it was highly popular among rationalists
- invented or contributed to various other potentially highly consequential ideas like UDASSA, acausal trade, influencing larger universes (which I noticed on Lightcone’s home page the other day), AI philosophical and strategic competence as x-risk factors, illegible AI safety problems
- has been doing all of this without ever being paid, as “charity” for the world
You’re saying this person has no natural home today on the modern Internet, and this is good and fine. You’re not very worried that maybe the kind of epistemic environment I “grew up” in had something to do with the intellectual habits that I developed, and you might be responsible for removing the last such place thereby “killing off” future Wei Dai’s.
What links here?
- Wei Dai's comment on Wei Dai’s Shortform by Wei Dai (24 Aug 2026 7:57 UTC; 15 points)
- Wei Dai's comment on Habryka’s Shortform Feed by habryka (23 Aug 2026 10:26 UTC; 5 points)