Wei Dai

Karma: 45,610

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 26 Mar 2026 0:34 UTC
7 points
0
in reply to: Max H’s comment on: ricraz’s Shortform
Another strain of thought from the early days of OB/LW is that the only or main alternative to a Singleton in the long run are Malthusian scenarios. I remember writing Non-Malthusian Scenarios (2009) to push back against this, but looking at it now, most of the non-Malthusian/non-Singleton scenarios aren’t actually that plausible or attractive.

Wei Dai 25 Mar 2026 1:03 UTC
5 points
3
in reply to: Richard_Ngo’s comment on: China Derangement Syndrome
One core premise of liberalism is that if you’re able to productively generate wealth in a non-aggressive way, then others shouldn’t interfere with that.
By “non-aggressive” are you only talking about external aggression (i.e., towards other countries) or also making a claim about internal repression (similar to cousin_it’s claim that it’s no worse than the US)?

Wei Dai 24 Mar 2026 22:09 UTC
7 points
2
on: The American Dream Is Alive In China
6 years later, this is looking too optimistic. Some subsequent (or not visible/salient at the time) governance failures:
1. Failing to pivot on zero-COVID policy, after Omicron made clear that infection rate and hence policy cost would be much higher.
2. Making real-estate a too large part of the economy, and housing costs too high (in part to fund government through higher land sale revenue). Subsequent RE bust causing large decreases in household wealth.
3. Various policies contributing to “involution” or excessive competition, leading e.g. 14 hour school days for 13yos, and overcapacity/loss-making in many industrial sectors.
4. Holding one-child policy for too long, which together with 2 and 3 (and other factors) causing TFR to drop to 1.02.
5. Foreign policy failures, e.g. wolf warrior diplomacy, Belt and Road Initiative.
6. Xi consolidating power, making subsequent policy failures more likely, due to fewer opportunities/incentives for criticism and debate.

Wei Dai 24 Mar 2026 20:39 UTC
1 point
−1
in reply to: cousin_it’s comment on: China Derangement Syndrome
likewise should be compared to the market system, where lots of people I talk to today feel like they can’t afford a child at all. I think when markets cause such bad outcomes, the market system should be held to the same standard as an autocratic system if it’d caused such outcomes; the market is not a magical blame-washing device to me.
I think this makes sense, but if you apply it to “affording a child” China is doing way worse in this regard (1.02 TFR vs 1.62 for the US).
“How many people the state imprisons or kills” summarizes many objective things and cuts through a lot of propaganda, and it gives the right answer for the past systems that we consider especially bad (like Mao’s China), so I think it’s probably accurate for today’s systems as well.
But any single measure like this can be really noisy. For example it surprised me to learn that incarceration rate for Taiwan is 236/100k vs 36 for Japan (165 for China, 541 for USA, 103 for South Korea), and I think you’ll be hard pressed to find anyone who thinks Taiwan is even 2 times more repressive than Japan, much less 6. (BTW the incarceration rate for Asians in the US is apparently 72.)

Wei Dai 24 Mar 2026 15:34 UTC
7 points
1
in reply to: cousin_it’s comment on: China Derangement Syndrome
Have you looked into:
- Financial repression (disposable income as % of GDP, limits on currency conversion)
- Censorship level (% of movies shown in theaters that are pre-approved by government censors, % of foreign websites that are accessible)
- Population/movement controls (not sure how to measure these, but e.g. “one-child policy” and hukou household registration system, which have no counterparts in the US)

Wei Dai 22 Mar 2026 20:39 UTC
LW: 3 AF: 3
−1
AF
in reply to: Steven Byrnes’s comment on: “Act-based approval-directed agents”, for IDA skeptics

But Jesus is long gone, and the cartoon character doesn’t even exist at all, and certainly neither was able to evaluate string theory ideas. They’re not “supervising” me in that sense.

Oh I didn’t realize this was your main point. To connect this to my most salient problem, namely how to improve production of philosophy and long-term strategy, I can’t think of anyone who is working in these areas and primarily motivated by the imagined approval of fictional or historical characters. Instead I think they’re mainly trying to win approval of other actual humans.

Do you think that nevertheless fictional approval (is this a good phrase to describe your idea?) is a promising avenue to pursue, for humans and/or AIs? A potential problem is that I don’t see how to ground it, i.e., if the imagined approval diverges from what’s actually good, there is no feedback loop to correct it?

But I do think there is some “…existence proof…” argument that goes through. E.g. at least some humans are making the overall situation better not worse (or if not, then we’re screwed no matter what), and AGIs don’t have to match the human population distribution.

It occurs to me that “at least some humans are making the overall situation better not worse” could be true, but a necessary factor is the constraints those humans have, e.g. limited intelligence, which can’t be reproduced in AIs. (If you limit your AI’s intelligence to make it safer / more aligned, someone will just copy your design and remove the limit.) E.g., maybe if I had von Neumann level IQ, I’d also be working in easy-to-verify domains like math and computer hardware, instead of philosophy and long-term strategy.

Wei Dai 22 Mar 2026 8:33 UTC
LW: 4 AF: 4
0
AF
in reply to: Steven Byrnes’s comment on: “Act-based approval-directed agents”, for IDA skeptics

I don’t think this is related to the points I was making in the post …

To spell out the relevance that I see, if the same “alignment tax” issue that you mentioned for approval-directed AIs occurs in humans, that means we can’t use humans as an “existence proof” that this problem is solvable, while at the same time if somebody was to come up with a solution to the problem for AIs, the same solution could plausibly be “back-ported” to humans and allow the smartest humans to be more productive in some especially important fields (like philosophy and long-horizon strategy).

Wei Dai 20 Mar 2026 23:03 UTC
2 points
0
in reply to: cousin_it’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution

Like, imagine we have a corrigible AI. Then a person using it can go off track very easily, by using the AI to help modify the AI and the person in tandem. To prevent that, the corrigible AI needs to have a lot of alignment-type stuff (don’t manipulate the user, don’t mislead, don’t go down certain avenues, what’s good what’s bad, etc) and that’s not too much different from having a moral AI.

Suppose corrigible AI is not very good at being aligned in this sense, I think both 1 and 3 are very bleak but see a bit more hope in 3 being able to navigate the situation better in some ways, by the humans being empowered being a bit smarter and more rational on average, and having fewer members to coordinate (when needed to avoid racing to the bottom in various ways).

(Also your earlier comment said “I’d much rather have 1”, which I was reacting to, so if you now only have a weak preference for 1 over 2 and 3, then I think we have much less of a disagreement.)

Wei Dai 18 Mar 2026 21:14 UTC
LW: 9 AF: 3
1
AF
on: “Act-based approval-directed agents”, for IDA skeptics

Second, there’s a capabilities (“alignment tax”) issue. The human supervisor is out in the environment in (c), but the (learned imitation of the) human supervisor is brought inside the AI’s thought process in (d). So in (d), much more than (c), we seem to be deeply constrained by the human supervisor’s competence and knowledge.

Is this why the smartest humans (e.g. John von Neumann, Terrance Tao) go into math, where verification is definitely easier than generation, instead of fields like philosophy and long-horizon strategy, where plans and outputs are much harder to judge by others? (JvN did do some philosophy and strategy, but surprisingly little relative to his abilities and interests, and I note that his philosophical work, in decision theory, was heavily math flavored.)

Wei Dai 18 Mar 2026 20:57 UTC
2 points
0
in reply to: cousin_it’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution
My grandparent comment was mostly addressing 1 vs 3, but I think 2 vs 3 is also very unclear. In order to make a moral AI, we also need to make it philosophically competent, and that seems like a hard problem, whereas with 3 we have fairly strong evidence that at least some humans or groups of humans can make philosophical progress over time, and some hope that this capability would be preserved by a corrigible AI + humans setup.

(I guess this is all assuming that there isn’t a long pause that allows AI philosophical competence or moral philosophy to be fully solved. Let me know if you’re talking about something else, e.g., what kind of AI we would ideally build after a long reflection.)

Wei Dai 17 Mar 2026 23:44 UTC
7 points
1
in reply to: cousin_it’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution
- First-best: some miracle happens and humans get a relatively equitable distribution of power. (Strawman scenario: everyone gets some amount of AI compute installed in their collarbone and can’t get more, and non-human AI capital winks out of reality.)
- Second-best: moral AIs win a decent chunk of the future.
- Third-best: corrigibility wins. AIs merge with human corporations / governments / rich individuals, and the resulting powerful entities own the future, screwing most people over.
I think you’re putting too much emphasis on “power corrupts” or power disparities as the main or only “human safety problem”, whereas I see a larger number of interlocking problems including lack of strategic competence, and having scary moral dynamics both in the face of AI and in “normal” circumstances.

This makes me more uncertain/ambivalent between these 3 options. I can see how each of them might turn out to be better than the others. Most relevant to the thread here, I think 3 might be a bit more rational as whole and a bit less prone to collectively going crazy under AI-powered manipulation / memetic evolution.

Wei Dai 17 Mar 2026 21:10 UTC
12 points
2
in reply to: Zack_M_Davis’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution

The powerful don’t need the powerless now, so why are they still alive? It’s not just that 15% of the government’s budget goes to welfare. The state barely even kills criminals anymore! Why?

I think it’s because we’re still in a democracy, and the voters prefer welfare/leniency partly as insurance (safety net for themselves), partly out of empathy/altruism.

A related question is why we’re still in a democracy, instead of plutocracy or oligarchy, and it occurs to me that plutocracy/oligarchy doesn’t seem very stable in our current world, e.g., both China and Russia were like that and then power became concentrated in the top ruler. So that could explain why the rich and powerful don’t have a strong incentive (yet) to push towards more concentration of power.

Wei Dai 15 Mar 2026 23:36 UTC
17 points
4
on: On The Independence Axiom
And just as with Euclid’s fifth, I believe, the resolution is not to keep trying harder to justify it but to ask: what happens when we drop it? What does the resulting decision theory look like? Is it consistent? Is it useful? Does it perhaps describe actual rational behavior better?
UDT was (in part) the result of asking this. See my 2009 indexical uncertainty and the Axiom of Independence, which pointed out that indexical uncertainty does not satisfy the Axiom of Independence, and in the comments @Vladimir_Nesov pointed out that this extends to more general kinds of uncertainty. Then UDT was written up 2 months later informed by this discussion.
(That write-up still used expected utility maximization with regard to logical/mathematical uncertainty, with a parenthetical note “This specifically assumes that expected utility maximization is the right way to deal with mathematical uncertainty. Consider it a temporary placeholder until that problem is solved.” AFAIK that problem is still open today. @Scott Garrabrant I think this answers your “weird to have it be formalized in term of expected utility maximization”?)

Wei Dai 13 Mar 2026 0:16 UTC
7 points
2
in reply to: Richard_Ngo’s comment on: Economic efficiency often undermines sociopolitical autonomy
I think the formal analysis of these concepts in economics is far from adequate (I haven’t deliberately kept up with the literature but expect that if there has been significant advances I would have heard about them), but that’s probably reflective of the actual difficulty of fully grounding these ideas in micro foundations (like game theory), not lack of attention or bad incentives among economists. (In other words I think you’ll probably have a hard time trying to do this yourself, unless you’ve got some promising novel approach that they’ve missed.)

In practice I mostly depend on my intuitions on these topics, which is partly informed by the formal/econ analysis I’ve seen, but perhaps you’re right that they’re “more like the sociopolitical versions of these concepts”.

Wei Dai 12 Mar 2026 23:09 UTC
3 points
0
in reply to: Richard_Ngo’s comment on: Wei Dai’s Shortform

That is: the scarier morality seems, the more you should suspect that your conception of morality is actually flawed.

I’m having more trouble understanding this comment than the parent one. Can you please expand on this more? Note that I’m quite confused/uncertain about what morality is; see Six Plausible Meta-Ethical Alternatives for an explanation of this.

Wei Dai 12 Mar 2026 23:05 UTC
31 points
0
on: Economic efficiency often undermines sociopolitical autonomy
Meanwhile, an example of the polar opposite strategy was Wei Dai declining to invest in Anthropic for moral reasons, thereby losing out on what would by now have been over 400x returns. I respect Wei’s approach very much (despite not knowing whether he should have been more econ-brained in this case).
This kind of echoes other people’s comments about how economics has incorporated a lot of sociopolitical thinking since the 1970s, but I would say I was pretty “econ-brained” when making this decision, specifically using concepts like
- signaling—not wanting to signal that I approved of Anthropic’s plans
- reputation—wanting to preserve my reputation as someone who cares a lot and thinks carefully about the future of our civilization and the universe
- utility function as subjective value—I’m allowed to consider my “non-economic” / subjective / intrinsic (e.g. moral) values
- bias / behavioral economics—COI potentially affecting my subsequent judgments
which are all now part of mainstream economics.
Edit: BTW @ESRogs pointed out that the actual return would be closer to 100x due to dilution.

Wei Dai 11 Mar 2026 5:05 UTC
2 points
0
in reply to: Chris Lakin’s comment on: Open Global Investment as a Governance Model for AGI
I answered this here previously. (Let me know if you’re asking a different question.)

Wei Dai 9 Mar 2026 17:10 UTC
3 points
0
in reply to: anaguma’s comment on: Wei Dai’s Shortform
I think so, but I’m a bit less confident. Philosophical competence and long-horizon strategic competence seems pretty correlated. I notice that that people who are most interested in long-horizon strategy all seem to be very interested in philosophy as well (e.g. Nick Bostrom, Eliezer Yudkowsky, Will MacAskill, myself). Philosophy can also directly increase strategic competence via improvements in decision theory, so increasing the philosophical competence of a misaligned AI may allow it to, for example, do more damage to the multiverse (and us) via acausal threats.

Wei Dai 8 Mar 2026 23:14 UTC
6 points
0
in reply to: Eli Tyre’s comment on: Wei Dai’s Shortform
I remember attending an academic cryptography conference when the Internet was just getting started (HTTP was just invented) and there were already hundreds of researchers there, with most top universities having at least a few. If you asked me back then, I would have said that by the analogous point in the Singularity, surely we’d have at least as many researchers in Singularity Strategies (or something like that).

Wei Dai 8 Mar 2026 22:11 UTC
2 points
0
in reply to: cdt’s comment on: Wei Dai’s Shortform

Whose values are you referring to here? Shared human values or particular values?

I guess my values, and other people’s values to the extent that I should care about them (with “should” having a to-be-determined meaning, pending a solution to metaethics).

Are you implying moral realism when you say this? (not a critique, just trying to follow.) Thanks.

No, see my metaethical position/uncertainty in this post.