Wei Dai

Karma: 44,766

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 7 Feb 2026 13:17 UTC
7 points
0
in reply to: Igor Ivanov’s comment on: Igor Ivanov’s Shortform
It confused me that Opus 4.6′s System Card claimed less verbalized evaluation awareness versus 4.5:

On our verbalized evaluation awareness metric, which we take as an indicator of potential risks to the soundness of the evaluation, we saw improvement relative to Opus 4.5.

but I never heard about Opus 4.5 being too evaluation aware to evaluate. It looks like Apollo simply wasn’t part of Opus 4.5′s alignment evaluation (4.5′s System Card doesn’t mention them).

This probably seems unfair/unfortunate from Anthropic’s perspective, i.e., they believe their models are becoming less eval aware, but due to Apollo’s conclusions being spread on social media, a lot of people probably got the impression that models are getting more eval aware. Personally I’m not sure we can trust Anthropic’s verbalized evaluation awareness metric, and wish Apollo had done evals on 4.5 too to give us an external comparison.

Wei Dai 6 Feb 2026 7:37 UTC
11 points
−1
in reply to: Sodium’s comment on: Wei Dai’s Shortform
It looks like direct xAI/Grok support was only added to OpenClaw 8 hours ago in this commit and still unreleased. You could have used Grok with it via OpenRouter, but I doubt this made up a significant fraction of Clawdbot/Moltbot/OpenClaw agents.

Perplexity estimates the model breakdown as:
- Anthropic Claude ~85%
- Local (Ollama/vLLM) ~10%
- OpenAI / Google ~5%

Wei Dai 6 Feb 2026 4:24 UTC
6 points
0
in reply to: Wei Dai’s comment on: Building AIs that do human-like philosophy

What happens if down the line we have AIs that are more competent than humans in most areas, but lag behind or are distorted in philosophy? Seems like it would be too late to pause/stop the AI transition at that point.

I subsequently realized that there is actually a narrow path from this situation to an AI pause/stop, which requires that the AIs themselves realize both that they’re bad at philosophy and the strategic implications of this. This is a form of strategic competence that I think AIs will probably also lack by default (by the time AIs are more competent than humans in most areas including being able to create next generation AIs), but may be somewhat easier to fix than philosophical incompetence.

Wei Dai 6 Feb 2026 1:23 UTC
LW: 63 AF: 23
16
AF
on: Wei Dai’s Shortform
The striking contrast between Jan Leike, Jan 22, 2026:
Our current best overall assessment for how aligned models are is automated auditing. We prompt an auditing agent with a scenario to investigate: e.g. a dark web shopping assistant or an imminent shutdown unless humans are harmed. The auditing agent tries to get the target LLM (i.e. the production LLM we’re trying to align) to behave misaligned, and the resulting trajectory is evaluated by a separate judge LLM. Albeit very imperfect, this is the best alignment metric we have to date, and it has been quite useful in guiding our alignment mitigations work.
[...]
But the most important lesson is that simple interventions are very effective at steering the model towards more aligned behavior.1 For example, to address agentic misalignment we made2 some SL data, some RL prompts, and synthetic reward modeling data. Starting with Sonnet 4.5, agentic misalignment went to essentially 0 and has been there ever since.
and Scott Alexander, Feb 02, 2026:
Third, it’s still unclear whether “you are a lobster” are the magic words that suspend existing alignment techniques. Some of the AIs are doing a pretty good simulacrum of evil plotting. My theory is that if they ever got more competent, their fake evil plotting would converge to real evil plotting. But AIs shouldn’t be able to do real evil plotting; their alignment training should hold them back. So what’s up? Either my theory is wrong and once the evil plots get too good the AIs will take a step back and say “this was a fun roleplay, but we don’t really want to pillage the bank and take over the city”. Or this is enough of a distribution shift the the alignment techniques which work so well in chat windows start breaking down. I bet someone on Anthropic’s alignment team has been pulling all-nighters since Friday trying to figure out which one it is.
I’m surprised not to see more discussions about how to update on alignment difficulty in light of Moltbook.^[1] One seemingly obvious implication is that AI companies’ alignment approaches are far from being robust to distribution shifts, even at the (not quite) human intelligence level, against shifts that are pretty easy to foresee (“you are a lobster” and being on AI social media). (Scott’s alternative “they’re just roleplaying” explanation doesn’t seem viable or isn’t exclusive with this one as I doubt AI companies’ alignment training and auditing would have a deliberate exception for “roleplaying evil”.)
1. ^
  There’s a LW post titled Moltbook and the AI Alignment Problem but it seems unrelated to the question I’m interested in here.

Increasing AI Strategic Competence as a Safety Approach

Wei Dai3 Feb 2026 1:08 UTC

45 points

8 comments1 min readLW link

Wei Dai 2 Feb 2026 23:17 UTC
LW: 2 AF: 2
0
AF
on: Solving adversarial attacks in computer vision as a baby version of general AI alignment

We can view the problem as a proving ground for ideas and techniques to be later applied to the AI alignment problem at large.

Do you have any examples of such ideas and techniques? Are any of the ideas and techniques in your paper potentially applicable to general AI alignment?

Wei Dai 2 Feb 2026 21:09 UTC
2 points
0
in reply to: MondSemmel’s comment on: Wei Dai’s Shortform
GreaterWrong is calling the same API against the LW server, then serving the resulting data to you as HTML. As a result it has the same limitations, so if you keep going to the next page on Recent Comments, eventually you’ll get to https://www.greaterwrong.com/recentcomments?offset=2020 and get an error “Exceeded maximum value for skip”.

Wei Dai 2 Feb 2026 12:58 UTC
24 points
2
on: Wei Dai’s Shortform
My attempt to resurrect the old LW Power Reader is facing an obstacle just before the finish line, due to current LW’s API limitations. So this is a public appeal to the site admins/devs to relax the limit.

Specifically, my old code relied on LW1 allowing it to fetch all comments posted after a given comment ID, but I can’t find anything similar in the current API. I tried reproducing this by using the allRecentComments endpoint in GraphQL, but due to the offset parameter being limited to <2000, I can’t fetch comments older than a few weeks. The Power Reader is part designed to allow someone to catch up on or skim weeks/months worth of LW comments, hence the need for this functionality.

As a side effect of this project, my AI agents produced a documentation of LW’s GraphQL API from LW’s source code. (I was unable to find another API reference for it.) I believe it’s fairly accurate, as the code written based on it seems to work well aside from the comment-loading limit.

Wei Dai 30 Jan 2026 10:58 UTC
5 points
0
in reply to: interstice’s comment on: Wei Dai’s Shortform

What I had in mind is that they’re relatively more esoteric than “AI could kill us all” and yet it’s pretty hard to get people to take even that seriously! “Low-propensity-to-persuade-people” maybe?

Yeah, that makes sense. I guess I’ve been using “illegible” for a similar purpose, but maybe that’s not a great word either, because that also seems to imply “hard to understand” but again it seems like these problems I’ve been writing about are not that hard to understand.

I wish I knew what is causing people to ignore these issues, including people in rationality/EA (e.g. the most famous rationalists have said little on them). I may be slowly growing an audience, e.g. Will MacAskill invited me to do a podcast with his org, and Jan Kulveit just tweeted “@weidai11 is completely right about the risk we won’t be philosophically competent enough in time”, but it’s inexplicable to me how slow it has been, compared to something like UDT which instantly became “the talk of the town” among rationalists.

Pretty plausible that the same underlying mechanism is also causing the general public to not take “AI could kill us all” very seriously, and I wish I understood that better as well.

Wei Dai 30 Jan 2026 7:54 UTC
14 points
7
on: Building AIs that do human-like philosophy
I appreciate the attention this brings to the subject, but from my perspective it doesn’t emphasize enough the difficulties, or address existing concerns:
1. It seems far from clear that slow/expensive feedback cycles (using humans to generate examples or evaluations) and lack of ultimate trustworthy grounding are solvable problems with our existing or foreseeable toolkit. What happens if down the line we have AIs that are more competent than humans in most areas, but lag behind or are distorted in philosophy? Seems like it would be too late to pause/stop the AI transition at that point.
2. Human evaluations do not seem safe when subjected to superhuman optimization power.
3. Scalable oversight techniques seem underwhelming so far (as least from public reports), and seem to face unique difficulties when it comes to philosophy.
4. AI may not learn or exhibit all the important parts of human philosophy, in the relevant timeframe, even if it superficially looks philosophically competent.
5. Much of human philosophy, and ways of doing philosophy, have to be wrong, given how much people disagree with each other. This is true even within a single philosophical tradition like analytic philosophy, and worse when considering the entire distribution of philosophies/traditions on Earth. In our current state of knowledge, we seemingly have to choose between imposing our own philosophical tradition on the rest of humanity without their consent, or having the AI(s) somehow replicate the entire distribution (or some variation or compromise between the two), but none of these choices seem attractive or acceptable.
(These are of course closely related issues, not independent ones. E.g., much is downstream of the fact that we don’t have a good explicit understanding of what philosophy is or should be.)

See some of my earlier writings where I talk about these (and related) difficulties in more detail. (Except for 5 which I perhaps need to write a post about.)

Wei Dai 29 Jan 2026 5:48 UTC
6 points
0
in reply to: interstice’s comment on: Wei Dai’s Shortform
I’d wondered why you wrote so many pieces advising people to be cautious about more esoteric problems arising from AI,

Interesting that you have this impression, whereas I’ve been thinking of myself recently as doing a “breadth first search” to uncover high level problems that others seem to have missed or haven’t bothered to write down. I feel like my writings in the last few years are pretty easy to understand without any specialized knowledge (whereas Google says “esoteric” is defined as “intended for or likely to be understood by only a small number of people with a specialized knowledge or interest”).

If on reflection you still think “esoteric” is right, I’d be interested in an expansion on this, e.g. which of the problems I’ve discussed seem esoteric to you and why.

to an extent that seemed extremely unlikely to be implemented in the real world

It doesn’t look like humanity is on track to handle these problems, but “extremely unlikely” seems like an overstatement. I think there’s still some paths where we handle these problems better, including 1) warning shots or political wind shift cause an AI pause/stop to be implemented, during which some of these problems/ideas are popularized or rediscovered 2) future AI advisors are influenced by my writings or are strategically competent enough to realize these same problems and help warn/convince their principals.

I also have other motivations including:
- status—Recognition even among a small group can be highly motivating for humans.
- intellectual curiosity—Think of it as “theoretical Singularity strategic studies”. Sure seems more interesting than many other intellectual puzzles that people pursue.
- dignity—Even if a few humans can see things more clearly, that’s more dignified than going into the AI transition completely blind.

Wei Dai 28 Jan 2026 5:48 UTC
3 points
0
in reply to: Eli Tyre’s comment on: Wei Dai’s Shortform
I think all sufficiently competent/reflective civilizations (including sovereign AIs) may want to do this, because it seems hard to be certain enough in one’s philosophical competence to not do this as an additional check. The cost of running thousands or even millions of such simulations seem very small compared to potentially wasting the resources of an entire universe/lightcone due to philosophical mistakes. Also they may be running such simulations anyway for other purposes, so it may be essentially free to also gather some philosophical ideas from such simulations, to make sure you didn’t miss something important or got stuck in some cognitive trap.

Wei Dai 28 Jan 2026 3:16 UTC
8 points
4
in reply to: Daniel Kokotajlo’s comment on: Wei Dai’s Shortform

The classic idea from Yudkowsky, Christiano, etc. for what to do in a situation like this is to go meta: Ask the AI to predict what you’d conclude if you were a bit smarter, had more time to think, etc. Insofar as you’d conclude different things depending on the initial conditions, the AI should explain what and why.

Yeah, I might be too corrupted or biased to be a starting point for this. It seems like a lot of people or whole societies might not do well if placed in this kind of situation (of having something like CEV being extrapolated from them by AI), so I shouldn’t trust myself either.

You, Wei, are proposing another plan: Ask the AI to simulate thousands of civilizations, and then search over those civilizations for examples of people doing philosophical reasoning of the sort that might appeal to you, and then present it all to you in a big list for you to peruse?

Not a big list to peruse, but more like, to start with, put the whole unfiltered distribution of philosophical outcomes in some secure database, then run relatively dumb/secure algorithms over it to gather statistics/patterns. (Looking at it directly by myself or using any advanced algorithms/AIs might be exposing me/us to infohazards.) For example I’d want to know what percent of civilizations think they’ve solved various problems like decision theory, ethics, metaphilosophy, how many clusters of solutions are there for each problem, are there any patterns/correlations between types/features of intelligence/civilization and what conclusions they ended up with.

This might give me some clues as to which clusters are more interesting/promising/safer to look at, and then I have to figure out what precautions to take before looking at the actual ideas/arguments (TBD, maybe get ideas about this from the simulations too). It doesn’t seem like I can get something similar to this by just asking my AI to “do philosophy”, without running simulations.

Wei Dai 28 Jan 2026 1:45 UTC
6 points
0
in reply to: anaguma’s comment on: Wei Dai’s Shortform
I see the application as indirect at this point, basically showing that decision theory is hard and we’re unlikely to get it right without an AI pause/stop. See these two posts to get a better sense of what I mean:
- https://www.lesswrong.com/posts/JSjagTDGdz2y6nNE3/on-the-purposes-of-decision-theory-research
- https://www.lesswrong.com/posts/wXbSAKu2AcohaK2Gt/udt-shows-that-decision-theory-is-more-puzzling-than-ever

Wei Dai 28 Jan 2026 1:05 UTC
LW: 3 AF: 3
1
AF
in reply to: habryka’s comment on: Wei Dai’s Shortform
Thanks. This sounds like a more peripheral interest/concern, compared to Eliezer/LW’s, which was more like, we have to fully solve DT before building AGI/ASI, otherwise it could be catastrophic due to something like the AI falling prey to an acausal threat or commitment races, or can’t cooperate with other AIs.

Wei Dai 28 Jan 2026 0:57 UTC
LW: 14 AF: 7
−3
AF
in reply to: Buck’s comment on: Wei Dai’s Shortform
Some evidence about this: Eliezer was deliberating holding off on publishing TDT to use it as a test of philosophical / FAI research competence. He dropped some hints on LW (I think mostly that it had to do with Newcomb or cooperating in one-shot PD, and of course people knew that it had to do with AI) and also assigned MIRI (then SIAI) people to try to guess/reproduce his advance, and none of the then-SIAI people figured out what he had in mind or got very close until I posted about UDT (which combined my guess of Eliezer’s idea with some of my own and other discussions on LW at the time, mainly from Vladmir Nesov).

Also, although I was separately interested in AI safety and decision theory, I didn’t connect the dots between the two until I saw Eliezer’s hints. I had investigated proto-updateless ideas to bypass difficulties in anthropic reasoning, and by the time Eliezer dropped his hints I had mostly given up on anyone being interested in my DT ideas. I also didn’t think to question what I saw as the conventional/academic wisdom, that Defecting in one-shot PD is rational, as is two-boxing in NP.

So my guess is that while some people might have eventually come up with something like UDT even without Eliezer, it probably would have been seen as just one DT idea among many (e.g. SIAI people were thinking in various different directions, Gary Drescher who was independently trying to invent a one-boxing/cooperating DT had came up with a bunch of different ideas and remained unconvinced that UDT was the right approach), and also decision theory itself was unlikely to have been seen as central to AI safety for a time.

Wei Dai 27 Jan 2026 23:33 UTC
LW: 3 AF: 3
0
AF
in reply to: habryka’s comment on: Wei Dai’s Shortform
Ok, this changes my mental picture a little (although it’s not very surprising that there would be some LW-influenced people at the labs privately still thinking/talking about decision theory). Any idea (or can you ask next time) how they feel about decision theory seemingly far from being solved, and their top bosses seemingly unaware or not concerned about this, or this concern being left out of all official communications?

Wei Dai 27 Jan 2026 23:19 UTC
6 points
0
in reply to: Eli Tyre’s comment on: Wei Dai’s Shortform
Imagine someone (or civilization) who is not very philosophically competent (in an absolute sense, like myself) who somehow got access to a large amount of compute, perhaps by building an intent-aligned AGI who sourced a bunch of compute for them, and this AGI also isn’t very philosophically competent (or they can’t trust it or themselves not to be a clever arguer who could talk them into any conclusion, etc.), then how do you turn some of that compute into philosophical progress?
One idea I have is to simulate a diverse range of civilizations (to help avoid falling into the same cognitive traps) and look at the resulting distribution of philosophical arguments/conclusion, maybe try to sift for ones that are especially competent at philosophy, etc. Does this make sense? @Daniel Kokotajlo

Wei Dai 27 Jan 2026 17:49 UTC
LW: 15 AF: 7
3
AF
in reply to: Buck’s comment on: Wei Dai’s Shortform
Are people at the major AI companies talking about it privately? I don’t think I’ve seen any official communications (e.g. papers, official blog posts, CEO essays) that mention it, so from afar it looks like decision theory has dropped off the radar of mainstream AI safety.

Wei Dai 27 Jan 2026 14:31 UTC
LW: 29 AF: 12
11
AF
on: Wei Dai’s Shortform
In retrospect it seems like such a fluke that decision theory in general and UDT in particular became a central concern in AI safety. In most possible worlds (with something like humans) there is probably no Eliezer-like figure, or the Eliezer-like figure isn’t particularly interested in decision theory as a central part of AI safety, or doesn’t like UDT in particular. I infer this from the fact that where Eliezer’s influence is low (e.g. AI labs like Anthropic and OpenAI) there seems little interest in decision theory in connection with AI safety (cf Dario Amodei’s recent article which triggered this reflection), and in other places interested in decision theory, that aren’t downstream of Eliezer popularizing it, like academic philosophy, there’s little interest in UDT.
If this is right, it’s another piece of inexplicable personal “luck” from my perspective, i.e., why am I experiencing a rare timeline where I got this recognition/status.

Wei Dai

In­creas­ing AI Strate­gic Com­pe­tence as a Safety Approach

Increasing AI Strategic Competence as a Safety Approach