Wei Dai

Karma: 45,549

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 18 Mar 2026 21:14 UTC
LW: 9 AF: 3
1
AF
on: “Act-based approval-directed agents”, for IDA skeptics

Second, there’s a capabilities (“alignment tax”) issue. The human supervisor is out in the environment in (c), but the (learned imitation of the) human supervisor is brought inside the AI’s thought process in (d). So in (d), much more than (c), we seem to be deeply constrained by the human supervisor’s competence and knowledge.

Is this why the smartest humans (e.g. John von Neumann, Terrance Tao) go into math, where verification is definitely easier than generation, instead of fields like philosophy and long-horizon strategy, where plans and outputs are much harder to judge by others? (JvN did do some philosophy and strategy, but surprisingly little relative to his abilities and interests, and I note that his philosophical work, in decision theory, was heavily math flavored.)

Wei Dai 18 Mar 2026 20:57 UTC
2 points
0
in reply to: cousin_it’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution
My grandparent comment was mostly addressing 1 vs 3, but I think 2 vs 3 is also very unclear. In order to make a moral AI, we also need to make it philosophically competent, and that seems like a hard problem, whereas with 3 we have fairly strong evidence that at least some humans or groups of humans can make philosophical progress over time, and some hope that this capability would be preserved by a corrigible AI + humans setup.

(I guess this is all assuming that there isn’t a long pause that allows AI philosophical competence or moral philosophy to be fully solved. Let me know if you’re talking about something else, e.g., what kind of AI we would ideally build after a long reflection.)

Wei Dai 17 Mar 2026 23:44 UTC
7 points
1
in reply to: cousin_it’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution
- First-best: some miracle happens and humans get a relatively equitable distribution of power. (Strawman scenario: everyone gets some amount of AI compute installed in their collarbone and can’t get more, and non-human AI capital winks out of reality.)
- Second-best: moral AIs win a decent chunk of the future.
- Third-best: corrigibility wins. AIs merge with human corporations / governments / rich individuals, and the resulting powerful entities own the future, screwing most people over.
I think you’re putting too much emphasis on “power corrupts” or power disparities as the main or only “human safety problem”, whereas I see a larger number of interlocking problems including lack of strategic competence, and having scary moral dynamics both in the face of AI and in “normal” circumstances.

This makes me more uncertain/ambivalent between these 3 options. I can see how each of them might turn out to be better than the others. Most relevant to the thread here, I think 3 might be a bit more rational as whole and a bit less prone to collectively going crazy under AI-powered manipulation / memetic evolution.

Wei Dai 17 Mar 2026 21:10 UTC
4 points
2
in reply to: Zack_M_Davis’s comment on: Terrified Comments on Corrigibility in Claude’s Constitution

The powerful don’t need the powerless now, so why are they still alive? It’s not just that 15% of the government’s budget goes to welfare. The state barely even kills criminals anymore! Why?

I think it’s because we’re still in a democracy, and the voters prefer welfare/leniency partly as insurance (safety net for themselves), partly out of empathy/altruism.

A related question is why we’re still in a democracy, instead of plutocracy or oligarchy, and it occurs to me that plutocracy/oligarchy doesn’t seem very stable in our current world, e.g., both China and Russia were like that and then power became concentrated in the top ruler. So that could explain why the rich and powerful don’t have a strong incentive (yet) to push towards more concentration of power.

Wei Dai 15 Mar 2026 23:36 UTC
14 points
2
on: On The Independence Axiom
And just as with Euclid’s fifth, I believe, the resolution is not to keep trying harder to justify it but to ask: what happens when we drop it? What does the resulting decision theory look like? Is it consistent? Is it useful? Does it perhaps describe actual rational behavior better?
UDT was (in part) the result of asking this. See my 2009 indexical uncertainty and the Axiom of Independence, which pointed out that indexical uncertainty does not satisfy the Axiom of Independence, and in the comments @Vladimir_Nesov pointed out that this extends to more general kinds of uncertainty. Then UDT was written up 2 months later informed by this discussion.
(That write-up still used expected utility maximization with regard to logical/mathematical uncertainty, with a parenthetical note “This specifically assumes that expected utility maximization is the right way to deal with mathematical uncertainty. Consider it a temporary placeholder until that problem is solved.” AFAIK that problem is still open today. @Scott Garrabrant I think this answers your “weird to have it be formalized in term of expected utility maximization”?)

Wei Dai 13 Mar 2026 0:16 UTC
7 points
2
in reply to: Richard_Ngo’s comment on: Economic efficiency often undermines sociopolitical autonomy
I think the formal analysis of these concepts in economics is far from adequate (I haven’t deliberately kept up with the literature but expect that if there has been significant advances I would have heard about them), but that’s probably reflective of the actual difficulty of fully grounding these ideas in micro foundations (like game theory), not lack of attention or bad incentives among economists. (In other words I think you’ll probably have a hard time trying to do this yourself, unless you’ve got some promising novel approach that they’ve missed.)

In practice I mostly depend on my intuitions on these topics, which is partly informed by the formal/econ analysis I’ve seen, but perhaps you’re right that they’re “more like the sociopolitical versions of these concepts”.

Wei Dai 12 Mar 2026 23:09 UTC
3 points
0
in reply to: Richard_Ngo’s comment on: Wei Dai’s Shortform

That is: the scarier morality seems, the more you should suspect that your conception of morality is actually flawed.

I’m having more trouble understanding this comment than the parent one. Can you please expand on this more? Note that I’m quite confused/uncertain about what morality is; see Six Plausible Meta-Ethical Alternatives for an explanation of this.

Wei Dai 12 Mar 2026 23:05 UTC
31 points
0
on: Economic efficiency often undermines sociopolitical autonomy
Meanwhile, an example of the polar opposite strategy was Wei Dai declining to invest in Anthropic for moral reasons, thereby losing out on what would by now have been over 400x returns. I respect Wei’s approach very much (despite not knowing whether he should have been more econ-brained in this case).
This kind of echoes other people’s comments about how economics has incorporated a lot of sociopolitical thinking since the 1970s, but I would say I was pretty “econ-brained” when making this decision, specifically using concepts like
- signaling—not wanting to signal that I approved of Anthropic’s plans
- reputation—wanting to preserve my reputation as someone who cares a lot and thinks carefully about the future of our civilization and the universe
- utility function as subjective value—I’m allowed to consider my “non-economic” / subjective / intrinsic (e.g. moral) values
- bias / behavioral economics—COI potentially affecting my subsequent judgments
which are all now part of mainstream economics.
Edit: BTW @ESRogs pointed out that the actual return would be closer to 100x due to dilution.

Wei Dai 11 Mar 2026 5:05 UTC
2 points
0
in reply to: Chris Lakin’s comment on: Open Global Investment as a Governance Model for AGI
I answered this here previously. (Let me know if you’re asking a different question.)

Wei Dai 9 Mar 2026 17:10 UTC
3 points
0
in reply to: anaguma’s comment on: Wei Dai’s Shortform
I think so, but I’m a bit less confident. Philosophical competence and long-horizon strategic competence seems pretty correlated. I notice that that people who are most interested in long-horizon strategy all seem to be very interested in philosophy as well (e.g. Nick Bostrom, Eliezer Yudkowsky, Will MacAskill, myself). Philosophy can also directly increase strategic competence via improvements in decision theory, so increasing the philosophical competence of a misaligned AI may allow it to, for example, do more damage to the multiverse (and us) via acausal threats.

Wei Dai 8 Mar 2026 23:14 UTC
6 points
0
in reply to: Eli Tyre’s comment on: Wei Dai’s Shortform
I remember attending an academic cryptography conference when the Internet was just getting started (HTTP was just invented) and there were already hundreds of researchers there, with most top universities having at least a few. If you asked me back then, I would have said that by the analogous point in the Singularity, surely we’d have at least as many researchers in Singularity Strategies (or something like that).

Wei Dai 8 Mar 2026 22:11 UTC
2 points
0
in reply to: cdt’s comment on: Wei Dai’s Shortform

Whose values are you referring to here? Shared human values or particular values?

I guess my values, and other people’s values to the extent that I should care about them (with “should” having a to-be-determined meaning, pending a solution to metaethics).

Are you implying moral realism when you say this? (not a critique, just trying to follow.) Thanks.

No, see my metaethical position/uncertainty in this post.

Wei Dai 8 Mar 2026 22:06 UTC
4 points
0
in reply to: interstice’s comment on: Wei Dai’s Shortform

But isn’t the relevant thing here whether solving some makes the overall situation worse in expectation?

It’s really unclear. On a meta level, it seems that we can’t trust our expectation estimates, due to tendency to ignore risks for status/power reasons. On the object level, there’s a very plausible mechanism why solving some problems can make things worse, and note that the choice isn’t between solving vs never solving some problem, but when to solve it.

Or maybe we should try to introduce new problems to further mitigate the set of existing problems!

That’s sort of what some people are trying to do, right, when they talk about bombing data centers, or doing hunger strikes in front of AI companies?

Wei Dai 7 Mar 2026 8:43 UTC
13 points
3
in reply to: Zack_M_Davis’s comment on: Wei Dai’s Shortform

If the future belongs to the best long-horizon planners,

If you’re not very strategically competent in an absolute sense, there are many ways to lose the future besides being taken over by a better long-horizon planner. For example you could build or fail to prevent/control some highly physically destructive technology, like grey goo or mirror life or doomsday nuclear device.

But I’m more worried about less legible risks, the most salient ones right now being that we lose our minds or values in some way (like AI psychosis but more powerful and widespread, or just the ordinary kind of cultural drift that Robin Hanson talks about), or derail philosophical progress (e.g. by building AIs that are bad at philosophical reasoning but good as persuasion, or lock in our values with AI help) such that we fail to eventually converge to correct conclusions about moral truths or our actual values.

If human-like cognitive skills necessarily came with long-horizon competence (which seemed plausible ten years ago) such that the kind of AI we have now would definitely already be planning to kill us, that would be a much worse situation.

Yes we’re lucky in this way, but what are we using our reprieve for? What can we use our reprieve for, that would actually lead to a good long term outcome? (My OP is basically saying “I don’t know” to this question.)

Wei Dai 7 Mar 2026 6:30 UTC
16 points
5
in reply to: StanislavKrym’s comment on: Wei Dai’s Shortform

What kind of instrumentally convergent goals could Tao even have and not act upon?

I mainly have in mind investigating AI x-risk, and then either helping mitigate it (it’s hard for me to imagine that a 230 IQ person can’t think of a way to help if they tried, at a minimum he could join those warning about it and calling for a pause or at least caution) or explain why it’s not worth his time to work on.

I struggle to even understand “long term implications of AI on philosophy production”

Here I mainly have in mind that AIs may be too good at persuasion relative to correct philosophical reasoning, able to convince us of nearly arbitrary philosophical conclusions, or otherwise derailing long term philosophical progress. I have some posts about this.

Wei Dai 7 Mar 2026 6:06 UTC
LW: 21 AF: 8
3
AF
on: Wei Dai’s Shortform
It seems that LLMs are not good enough at reasoning, even after being trained on ~all human output, such that you couldn’t amplify their capabilities to arbitrary levels through iterated amplification, so AI companies are mainly increasing AI capabilities via RLVR instead. Is this impression wrong, and how to update on it if not?
Aside from the potential implications on alignment (i.e., closing off one approach that seemed hopeful for some, at least for the foreseeable future), I wonder if this is a deficiency in LLMs (their architecture or how they’re trained), a deficiency in human reasoning (i.e., even HCH was never going to work), or a deficiency in reasoning itself (something like, there is no such thing as a reasoner that can be amplified to arbitrary capability levels purely through iteration)?

Wei Dai 7 Mar 2026 3:38 UTC
132 points
6
on: Wei Dai’s Shortform
Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. With very few exceptions, billionaires spend or give away their money haphazardly, philosophers don’t bother to think about long term implications of AI on philosophy production (positive or negative), Terence Tao spends his time wireheading on abstract math instead of doing anything remotely like instrumental convergence. Unlike my youthful expectations (upon reading Vernor Vinge), there are no university departments filled with super-geniuses charting a path for humanity to safely navigate the Singularity.

Aside from this, humans also have a bunch of other safety problems, like being bad at philosophy, being easy to manipulate, having strange and unstable values, tending to ignore risks they create (because acknowledging them would be bad for one’s status). So if you try to improve people’s agency, you likely just end up getting people like founders of FTX and OAI.

What about getting help from AI? Well they seem to suffer from many of the same safety problems, but in even more severe forms. E.g., current AI capabilities are even more skewed towards short-horizon, easily verifiable tasks, like math and coding. They seem even more prone to reward gaming, are even worse at doing philosophy, are liable to have even more alien values, etc.

Both AI and human safety seem to have this interlocking nature, i.e., there is a bunch of different safety problems where solving some but not all of them at the same time can make the overall situation worse. (For example, solving AI intent alignment allows humanity to do more damage to itself with AI help, if AI doesn’t also provide competent strategic and philosophical assistance, but increasing AI strategic competence risks allowing misaligned AI to take over more easily.) This feature demands a high level of strategic competence to recognize and navigate, which is just what we don’t have.

I’ve been supportive of AI pause/stop, to buy time for human intelligence amplification and/or AI safety research, but increasingly think even that’s not going to be sufficient to get a good long term future, because these activities, even if they succeed, would likely solve only some of the interlocking safety problems. For example, increasing human intelligence seems likely to increase our technical abilities more than our philosophical and strategic competence, and it is also risky in other ways due to human safety problems that nobody is working on, e.g., positional competition. Even a very long AI pause, e.g. thousands or millions of years, may not suffice because it’s not clear what dynamic would push humanity to eventually fix all of its safety problems at the same time, before it did something else irreversibly damaging.

I don’t have any good ideas for what to do in light of all this. Just wanted to post an update on my current thinking, my own “situational awareness”, if you will. (I guess I still support AI pause to some degree, just to kick the can down the road and buy some more time to think.)
What links here?
- Wei Dai's comment on Terrified Comments on Corrigibility in Claude’s Constitution by Zack_M_Davis (17 Mar 2026 23:44 UTC; 7 points)

Wei Dai 5 Mar 2026 23:55 UTC
4 points
1
in reply to: wassname’s comment on: Raemon’s Shortform Feed

One solution is to integrate a proof of humanity type ID

This doesn’t prevent humans from copy-pasting AI generated output, or (in the future) prevent AIs from hiring humans to post their writings. The post you linked to has a section “Can we prevent renting IDs (eg. to sell votes)?” but the ideas there do not seem to apply to the current use case.

Wei Dai 5 Mar 2026 23:41 UTC
5 points
1
in reply to: nielsrolf’s comment on: Raemon’s Shortform Feed
How do you prevent the poster from using AI to help them pass the quiz?

Wei Dai 5 Mar 2026 8:06 UTC
3 points
0
in reply to: E. P. Cooper’s comment on: Wei Dai’s Shortform
Thank you, I’ve taken your suggestion and removed all uses of Math.random().