Wei Dai

Karma: 45,182

If anyone wants to have a voice chat with me about a topic that I’m interested in (see my recent post/comment history to get a sense), please contact me via PM.

My main “claims to fame”:

Created the first general purpose open source cryptography programming library (Crypto++, 1995), motivated by AI risk and what’s now called “defensive acceleration”.
Published one of the first descriptions of a cryptocurrency based on a distributed public ledger (b-money, 1998), predating Bitcoin.
Proposed UDT, combining the ideas of updatelessness, policy selection, and evaluating consequences using logical conditionals.
First to argue for pausing AI development based on the technical difficulty of ensuring AI x-safety (SL4 2004, LW 2011).
Identified current and future philosophical difficulties as core AI x-safety bottlenecks, potentially insurmountable by human researchers, and advocated for research into metaphilosophy and AI philosophical competence as possible solutions.

My Home Page

Wei Dai 1 Mar 2026 5:09 UTC
LW: 2 AF: 2
0
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal
Please see the section entitled “This essay is not very skimmable”

I had seen that warning, and was trying to keep track of the distinction, but apparently still failed. To check my understanding now:
1. cosmic Schelling answers are hypothetical answers in thought experiments where we assume that everyone is trying to converge to the same answers on the same questions
2. cosmic Schelling norms are just a subset of cosmic Schelling answers (“to a pro tanto moral question”), and therefore not necessarily actual norms in the dictionary sense of “a principle of right action binding upon the members of a group and serving to guide, control, or regulate proper and acceptable behavior”
In other words, the cosmic Schelling norm of an arbitrary pro tanto moral question probably exists in platonic space, but in most cases this would not be an actual norm in reality because (among other potential reasons) most beings in the cosmos would not actually be trying to converge on this particular question. Is this correct?

(If so, I’m confused how this usage of “norm” squares with your position as a compositional language realist, since compositionally it seems like a statement of the form “X is a cosmic Schelling norm” should imply that X is a norm?)

Wei Dai 1 Mar 2026 0:54 UTC
LW: 2 AF: 2
0
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal

Is your point trying to use the same definitions and ontology as in the post, and responding to a particular logical argument made within it?

Yes to the former, no the latter. I was more trying to “sanity check” your overall approach against my intuitions and trying to explain why I feel very skeptical about it. But I can try to trace back what part of the post I start to disagree with:

Let’s talk about stealing as a concrete example, since we haven’t discussed that yet.

What’s the cosmic Schelling answer to the question, “Is stealing good or bad?”

I’m not sure if there’s an even earlier step, but here it jumps out at me that you seem to have chosen “stealing” as an example because it’s a highly salient moral question for humans, but it may well not be for other civilizations, like my hypothetical one. You seem to be implicitly assuming that everyone will try to converge on the same questions (because “cosmic Schelling question asks what answer beings would converge on when trying to converge on the same answer”, which seemingly wouldn’t make sense to do if others are not actually trying to converge on the same questions), whereas my intuition is that norms and coordination mechanisms in general may be highly context-dependent so this seems like an unjustified assumption.

Wei Dai 28 Feb 2026 18:05 UTC
LW: 0 AF: 2
0
AF
in reply to: Andrew_Critch’s comment on: Schelling Goodness, and Shared Morality as a Goal
I’m not sure how relevant this line of questioning is to my main point. (You might be focused too much on a part of my comment that isn’t all that load carrying.) As I wrote in a parallel thread:

Because sometimes it’s easier or more efficient to spin up a new agent with a known good state than to try to help one that has gone off rails. It’s also possible that for more advanced agents this will never or almost never be the case, in which case perhaps “stealing” just won’t be a commonly used or thought about concept in this kind of civilization. My main point is that “stealing is bad” being a salient idea seems quite contingent on some features of current humans and our civilization, so I’m skeptical of it being a scale-invariant Schelling point for a “cosmically general population”, and more generally skeptical that it makes sense to think about morality in this way.

Wei Dai 28 Feb 2026 15:53 UTC
3 points
0
in reply to: StanislavKrym’s comment on: Schelling Goodness, and Shared Morality as a Goal

Then, if one agent sees the other agent performing poorly, then why would the first agent decide not to help the second one with the task (e.g. by giving hints which prevent the mental process from circulating in the wrong region of ideas) instead of stealing resources?

Because sometimes it’s easier or more efficient to spin up a new agent with a known good state than to try to help one that has gone off rails. It’s also possible that for more advanced agents this will never or almost never be the case, in which case perhaps “stealing” just won’t be a commonly used or thought about concept in this kind of civilization. My main point is that “stealing is bad” being a salient idea seems quite contingent on some features of current humans and our civilization, so I’m skeptical of it being a scale-invariant Schelling point for a “cosmically general population”, and more generally skeptical that it makes sense to think about morality in this way.

Wei Dai 28 Feb 2026 13:57 UTC
LW: 10 AF: 3
−2
AF
on: Schelling Goodness, and Shared Morality as a Goal
When I’m doing “agentic coding”, I “kill” and “steal” from my agents all the time (i.e., terminate some coding agent that’s going off in a wrong direction and reclaim resources allocated to them), and my agents kill their sub-agents all the time.

Suppose a whole civilization was like internally like this (e.g., countless agents all aligned to some central superintelligence), and externally it took over their planet by conquest and/or negotiated mergers (under threat of war/conquest). I think such a civilization may well think “stealing is good”, “because requesting and waiting for permission before violating resource boundaries often cause a waste of resources” or it never arises as a moral issue at all.

The lesson from this seems to be that “stealing is bad” is contingent on:
1. Lack of alignment technology, leading to unavoidable value differences, which incentivizes stealing and makes it a moral issue in the first place.
2. Gains from stealing or taking resources by force are limited because you can’t spin up an aligned agent to make full use of them.

Wei Dai 27 Feb 2026 14:12 UTC
6 points
0
in reply to: Richard_Kennaway’s comment on: Richard_Kennaway’s Shortform
What about Tegmark Level I (copies of Earth existing far away from each other in infinite space) and the associated kind of immortality?

Wei Dai 23 Feb 2026 12:51 UTC
19 points
0
on: Changing the world for the worse
I posted my own take on this subject a while ago.

That I don’t want to try to have a big impact, if I can’t be certain that that impact will be positive rather than negative for the world — and I can’t be certain.

I’ve been doing something similar, in part because I seem to instinctively shy away from high-stakes situations. But I haven’t publicly endorsed this, because excessively dissuading people from attempting to have big positive impacts could itself constitute a big negative impact, and I’m not sure how to balance this. (Hence the “managing risks” framing in my post linked above, which seems “safer” from this perspective.)

Also, I make an exception for myself, namely that it’s ok to talk about my abstract ideas (which can cause big impacts if others eventually translate them into concrete ideas and actions), if I take sufficient precautions. This seems justified because if nobody talks about their ideas, world would be stuck in its current intellectual state, so there seems to be no other alternative.

Wei Dai 19 Feb 2026 12:15 UTC
5 points
0
in reply to: TsviBT’s comment on: Genomic emancipation contra eugenics
Are there one or two examples that are most compelling to you on this point?
- Chinese/Korean schools/parents giving their kids incredible amounts of homework/schoolwork. My understanding is that this is typical for a 13yo in China: “Hi guys, it’s 8:30pm, let’s go pick up Cindy together,” 42-year-old Ray Liu said. “She went to school this morning at 7 a.m., so she stayed at school for about 14 hours.”
- Fashions in clothing/decorating/design, causing people to throw out perfectly usable clothes and remodel “outdated” kitchens and bathrooms.
- Virtue signaling, which is inherently positional and also has a large “fashion” dynamic
- Wars and other forms of power competition such as palace intrigues and AI race.
Looking this over, I think my top worries related to “positional arms races” are now:
1. People pushing their kids’ genomes in “fashionable” directions, potentially causing a variety of problems, foreseeable (reduced diversity, fashions often being costly or harmful, value drift as people use their kids’ genes to virtue signal in random directions) and unforeseeable.
2. An “IA race” where people/countries take too much risk in order to gain power and/or status based on IA.

Wei Dai 19 Feb 2026 5:50 UTC
17 points
4
on: Genomic emancipation contra eugenics
Thanks for bringing up positional arms races (the “don’t make your sons 6′8″” example), which hadn’t previously occurred to me in the context of reprogenetics.
You identify the problem clearly, but your own commitments seem to rule out every plausible mechanism for actually preventing it:
- The state can’t intervene (genomic liberty)
- Counselors shouldn’t steer choices (neutrality)
- Society shouldn’t pressure specific choices (anti-soft-eugenics)
- Choices should be private (privacy norms)
That leaves moral exhortation to individual parents, and well, the many examples of positional arms races throughout human history suggest that this alone is unlikely to work.

Wei Dai 18 Feb 2026 21:11 UTC
2 points
0
in reply to: ESRogs’s comment on: Wei Dai’s Shortform
A 236 karma r/ClaudeCode post today:

Claude is dropping max plans for enterprise (maybe for everyone?)

Not sure if anyone else has seen this.

My company has our developers on max x20 plans. We were told that once our current contract was up everyone had to switch to pay-as-you-go api pricing. We prodded our rep and the response was basically that the max plans aren’t profitable so they’re getting rid of them.

From his tone it didn’t sound like he was just talking about enterprises. We’ve all known that Anthropic has been burning money, and wondering how long they can keep it up. My friends, I’m afraid the end may be nigh.

Wei Dai 18 Feb 2026 2:34 UTC
11 points
2
in reply to: Wei Dai’s comment on: Eli’s shortform feed
I found^[1] Eliezer’s 2012 comment where he talked about why he didn’t want FAI to solve philosophical problems for itself:
I have been publicly and repeatedly skeptical of any proposal to make an AI compute the answer to a philosophical question you don’t know how to solve yourself, not because it’s impossible in principle, but because it seems quite improbable and definitely very unreliable to claim that you know that computation X will output the correct answer to a philosophical problem and yet you’ve got no idea how to solve it yourself. Philosophical problems are not problems because they are well-specified and yet too computationally intensive for any one human mind. They’re problems because we don’t know what procedure will output the right answer, and if we had that procedure we would probably be able to compute the answer ourselves using relatively little computing power. Imagine someone telling you they’d written a program requiring a thousand CPU-years of computing time to solve the free will problem.
Interesting to compare this to my Some Thoughts on Metaphilosophy, where I argued for the opposite.
1. ^
  Using my recently resurrected LW Power Reader & User Archive userscript. The User Archive part allows one to create an offline archive (in browser storage) of someone’s complete LW content and then do a search like /philosoph/ replyto:Wei_Dai

Wei Dai 16 Feb 2026 20:12 UTC

4 points

in reply to: ESRogs’s comment on: Wei Dai’s Shortform

Have you seen these? While not directly talking about subscriptions vs API pricing, they suggest that overall gross economics look thin to negative which suggests the much (1-2 OoM) cheaper subscriptions are likely loss-making (if heavily used). The OAI article talks specifically about inference costs and argues inference spend alone appears to be greater than the revenues it compares against.

Perplexity’s (w/ GPT-5.2 Thinking) conclusions about how credible these articles are:

Link	What’s most solid	What’s least solid
“OpenAI inference spend”	Document-based claim + acknowledges missing revenue-share flows; FT involvement noted. wheresyoured	Implied revenue math can be incomplete given complex cross-payments; Microsoft reportedly says “not quite right.” wheresyoured+1
“Anthropic/Cursor AWS spend”	Specific AWS spend figures asserted as “final cash paid…after discounts/credits,” reportedly cross-verified with other sources. wheresyoured	Revenue side is heavily estimate-driven; big conclusions depend on assumptions about non-AWS spend and timing. wheresyoured

Wei Dai 16 Feb 2026 13:49 UTC
10 points
4
in reply to: Richard_Ngo’s comment on: Aligning to Virtues

Defer to the processes that produced the virtues we already have.

The way that cultural evolution happened in the past involved a lot of wars/conquests and people copying powerful cultures in part out of fear. What would “deferring to this process” look like going forward, or would you get rid of this part? What do you think about Robin Hanson’s concerns around “cultural drift”?

Specifically reason about the consequences of different virtues.

Do AIs also need to do this? If so, do we not still need to align them with consequentialist values?

Wei Dai 16 Feb 2026 9:34 UTC
17 points
8
on: Aligning to Virtues
I think in the past, virtues were the result of memetic/cultural evolution. How do you envision them being created or selected in the future? (The main idea that comes to my mind is some kind of consequentialist reasoning or computation, e.g. humans or AIs trying to figure out what virtues would lead to the best consequences, or doing large scale simulations to find this out. But this involves a dependency on consequentialism, which I guess you’d want to avoid?)

Wei Dai 16 Feb 2026 1:02 UTC
21 points
2
in reply to: Eli Tyre’s comment on: Eli’s shortform feed
Eliezer 2018

I do also remark that there are multiple fixpoints in decision theory. CDT does not evolve into FDT but into a weirder system Son-of-CDT. So, as with utility functions, there are bits we want that the AI does not necessarily generate from self-improvement or local competence gains.

2018 FB discussion

Michael Cohen

I think, with >99% prob., if we make an aligned superintelligent causal decision theorist, that we get an “existential win.” I take the “MIRI view” on pretty much every other point, so I’m more inclined than I otherwise would be in investigate the possibility of something I currently take to be very unlikely: that our lives depend on another decision theory. Is there anyone (perhaps someone at MIRI) who can explain the claim or point me to a link explaining why they expect an aligned causal decision theorist to fail? To kick off the discussion, I understand that at t=2, it’s not a causal decision theorist anymore, but I trust the updated agent too, if the causal decision theorist ceded power to it.

The relevance of my position would be to shift decision theory resources to ontology identification, naturalized induction, and generalizable environmental goals—things where I would pretty surprised if we make aligned AGI without understanding how to solve those problems.

Eliezer Yudkowsky

I think your first AGI is supposed to do some non-ambitiously-aligned boundedly-accomplishable task that causes the world to not be destroyed by the next, unaligned AGI built 3 months later. You could plausibly get away with using a CDT agent for this so long as the agent only thought about a bounded range of stuff near in space, time, and probability; was not generally superintelligent across all domains; and was not freely self-modifying so that it kept those properties.

Otherwise an LDT agent can take all of a CDT agent’s marbles, or take nearly all of the gains from trade across even positive-sum interactions, via e.g. the LDT agent predictably refusing any offer short of $9.99 in the Ultimatum Game. The CDT agent building the Son-of-CDT agent will likewise find, in a way that it thinks has nothing to do with its physical acts, that the LDT agent seems to have predictably configured itself to reject offers less than $9.99 from whatever next agent the CDT agent builds, so this exploitability is a reflectively stable property of the descendants of fully reflective CDT agents. This makes CDT unsuitable for ambitious alignment projects like CEV, but you shouldn’t be trying that anyways on your first AGI.

Michael Cohen

Suppose the agent is freely self-modifying and unbounded, and no other AGI exists yet. Can’t it do all this reasoning above and make sure it has replaced itself with an unexploitable AGI by the time any scary adversary comes along?

Eliezer Yudkowsky

No. A CDT agent predicts that an LDT agent will ignore its attempt to be unexploitable and go on rejecting offers short of $9.99. The problem is that the CDT agent is incapable of seeing controllability in the connection between its own abstract computation to accept the unfair offer in the Ultimatum Game, and the LDT agent having already simulated that computation and figured out how to exploit it.

The CDT replicates this property in its offspring because the CDT agent does not see a controllable connection between its computed choice to build an offspring like that and the LDT agent’s choice to reject offers short of $9.99 from its offspring. CDT agents don’t build LDT agents, they build weird reflectively consistent agents we label Son-of-CDT which continue to think that nothing is controllable unless it stems from a physical consequence of the CDT agent picking a particular computation. Tl;dr CDT agents lose all the precommitment wars and so do other agents build by CDT agents.

Eliezer Yudkowsky

A simple gloss on why you ought to understand LDT before building a CDT Task AGI is that this study tells us ways we need to restrict a CDT Task AGI’s cognition to prevent it from asploding.

Eliezer Yudkowsky

Or to put it another way, if you write the most sophisticated program you are smart enough to write, you are not smart enough to debug it. A corollary of this principle is that you ought to understand the theory of the program one step more sophisticated than the less sophisticated program you actually write, in order to understand all the tradeoffs you are making by writing the less sophisticated program. It’s fine if you say that a mergesort is simpler than a quicksort, but I’d much more trust somebody who understood quicksort who then said that they needed to write a mergesort because that was easier to debug, as opposed to somebody who didn’t understand quicksort saying that they were going to write a mergesort because that was the best sorting algorithm they thought they could grasp. You need conceptual error margin between what you think you could build and what you try to actually build, so that you understand what lies on both sides of the theory-boundary around your choice to use a theory at that specific level of complication.

Michael Cohen

I’m skeptical that “son-of-cdt” is something the future ldt can define well. Certainly it doesn’t mean any agent created after the cdt agent exists, because that would be all agents, and the fact that there is has been one human cdt in the past would have resolved the ldts behavior already. There are, of course, lots of ways for our cdt AGI to create new agents that are aligned with it. How is the ldt supposed to define whether the cdt AGI made this new agent for which it is wondering whether to treat like as a sucker? If we created an ldt AGI instead, but the human creator of the ldt AGI had a cdt ancestor, will the future ldt AGI only offer our ldt AGI one cent in the ultimatum game?

(Tangent: this is all from the intuition that most things shouldn’t depend on the continuity of the self, or whatever the term is for what parfit rejects. Continuity of the self seems like an even more dubious concept for AIs where “instances” of a “self” can branch in time rather than one succeeding the next with every passing second.)

tl;dr any action the cdt takes will change the expected number of agents aligned with it that don’t get bullied by the ldt in the future. Which cases are “its sons”?

Eliezer Yudkowsky

Son-of-CDT is a particular algorithm that it seems CDT agents should build in a pretty convergent way. Its properties are well-defined to the extent CDT is well-defined. Taking all of its marbles is as easy as taking all of CDT’s marbles.

Michael Cohen

I’m not sold on that. Son-of-CDT could be literally anything, and it depends on empirical facts, like what sort of adversaries are likely to exist in the future. Indeed, any agent that could be created in the interim between cdt-birth and ldt-birth will be considered by cdt, and if ldt treats any of them fairly, that’s the sort of agent that will appear. If none of them will be treated fairly, that means ldt gives this treatment to all agents born after a cdt has existed (and that ship has sailed).

Michael Cohen

A separate sufficient objection (and if my case rests on this I acknowledge a good deal of capitulation): if it’s the first agi and it’s unbounded, it will secure a decisive advantage and make sure that nothing that can challenge it ever gets made. EDIT: In fact, even if it is bounded, it only meets our standard for an existential win if it is able to ensure than nothing that can challenge it ever gets made. So once we’ve figured out how to do that, there are no additional gains from giving it a decision theory besides cdt.

Christopher Leong

What’s LDT and is there a paper on how LDT agents can run this exploit?

Eliezer Yudkowsky

Try Logical decision theories.

Linda Linsefors

A CDT agent predicts that an LDT agent will ignore its attempt to be unexploitable and go on rejecting offers short of $9.99. The problem is that the CDT agent is incapable of seeing controllability in the connection between its own abstract computation to accept the unfair offer in the Ultimatum Game, and the LDT agent having already simulated that computation and figured out how to exploit it.

You are assuming that the LDT agent knows the history of the son-of-CDT is. How did the LDT agent get this information?

The son-of-CDT agent will notice that if it can hide its history from the LDT agent, e.g. tricking it into believing that it was always LDT-like, or just making the LDT uncertain about the origin of the son-of-CDT, then, with the right self modifications, the son-of-CDT will not be exploitable.

Me (Wei Dai) 2019

One meta level above what even UDT tries to be is decision theory (as a philosophical subject) and one level above that is metaphilosophy, and my current thinking is that it seems bad (potentially dangerous or regretful) to put any significant (i.e., superhuman) amount of computation into anything except doing philosophy.

To put it another way, any decision theory that we come up with might have some kind of flaw that other agents can exploit, or just a flaw in general, such as in how well it cooperates or negotiates with or exploits other agents (which might include how quickly/cleverly it can make the necessary commitments). Wouldn’t it be better to put computation into trying to find and fix such flaws (in other words, coming up with better decision theories) than into any particular object-level decision theory, at least until the superhuman philosophical computation itself decides to start doing the latter?

(Current thoughts) I’m not sure why SIAI/MIRI didn’t take a position more similar to mine (which you may share, if in “Why wouldn’t an early seed AI reason about the ways that it’s decision theory makes it exploitable” you mean philosophical reasoning as opposed to “reasoning through its current decision theory”) that ideally the seed AI would solve decision theory for itself by doing good philosophy. Eliezer may have written one comment (that I can recall) where he said something about this, but given lack of good searching abilities I’ll have to spin up an AI-assisted project to find it.

Edit: Forgot to mention that I think the AI solving decision theory (and other philosophical problems) incorrectly, by doing bad philosophy, is a serious risk. Not so much that it would pick CDT or Son-of-CDT, but messing up more subtly, like putting oneself in a worse bargaining position by simulating another agent for too long (thereby giving the other agent an incentive to commit to only accepting a deal favorable to themselves). Or more generally not figuring out a solution to such commitment race problems, or the many other tricky decision theory problems.

Wei Dai 15 Feb 2026 13:49 UTC
4 points
0
in reply to: cousin_it’s comment on: Long-term risks from ideological fanaticism

The power imbalance was what made bad things happen. So my preferred interventions for the future would give people the power to fight back against bigger agents who would threaten them, ideology or no.

The problem with this is that power imbalances were also what made some important good things happen, so without (and if we don’t come up with a replacement mechanism) we can also get into big trouble. Robin Hanson has been talking about some aspects of this under “cultural drift”.

The most salient example (to me) of this is the spread of analytic philosophy. Without the British Empire (or the West as a whole) taking over or spreading its culture by force over large parts of the world (including by others imitating Western culture in hopes of avoiding conquest, like Japan, and China to a lesser extent), the number of people with a reasonable philosophical tradition would be even tinier than it is today.

(One could tell a plausible story about this, that good philosophy leads to good science which leads to greater military power. Or that power competition helps cull harmful memetic parasites.)

Of course this is overall terrible, and not something I endorse, but neither would I endorse an intervention to get rid of power imbalances, without figuring out how to solve this kind of problem first. (Also, to be clear, I do not think we should rest our hopes of future civilization being philosophical competent on such power dynamics. It seems like a very risky / non-robust way for that to happen, and my point is just that we could make things even worse by getting rid of it and not having a better alternative in place.)

Wei Dai 14 Feb 2026 15:06 UTC
4 points
0
in reply to: ChristianKl’s comment on: Wei Dai’s Shortform
It seems like a good question, but unfortunately I have no familiarity with superforecasting, having never learned about it or participated in anything related except by reading some superficial descriptions of what it is.

Until Feb 2020 I had little interest in making empirical forecasts, since I didn’t see it as part of my intellectual interests, and believed in EMH or didn’t think it would be worth my time/effort to try to beat the market, so I just left such forecasting to others and deferred to other people who seem to have good epistemics.

If I had to guess based on my shallow understanding of superforecasting, I would say while there are probably overlapping skills, there’s a strategic component to trading, which involves things like which sectors to allocate attention to, how to spot the best opportunities and allocate capital to them, while not taking too much concentrated risk, explore vs exploit type decisions, which are not part of superforecasting.

Wei Dai 14 Feb 2026 12:44 UTC
11 points
2
in reply to: Nick Bostrom’s comment on: Optimal Timing for Superintelligence: Mundane Considerations for Existing People
Nick, I’m afraid that a faction^[1] of your moral parliament may have staged a (hopefully temporary) coup or takeover, because if all of the representatives were still in a cooperative mood it seems like you’d probably have inserted at least a few more sentences to frame it differently to mitigate potential risks. You have enough people around you who would presumably be happy to help you with this even if you “have no comparative advantage” in it. (Comparative advantage is supposed to be an argument for trade, not an excuse for ignoring risks/downsides to your other values!)
1. ^
  perhaps a coalition of egoism, person-affecting altruism, and intellectual pursuit for its own sake

Wei Dai 14 Feb 2026 12:07 UTC
4 points
0
on: Wei Dai’s Shortform
Positional (e.g. status) competition isn’t literally zero-sum if different people have different “measures” or “magical reality fluid” in Eliezer’s parlance, which seems pretty plausible (e.g. due to simulations and/or something like UDASSA). This can be another reason for one’s moral parliament to endorse behavior that would be conventionally viewed as zero sum.
(The more common-sensical reason being that one’s positional drives/motivations/values ought to have some representatives in the moral parliament who get their way sometimes, i.e., when it doesn’t hurt the other values much.)
I’ve been meaning to make a post about this, and this discussion just reminded me to. Hopefully it’s immediately obvious once pointed out, so I’ll keep this short.
(After consulting a bunch of AIs, their consensus is that the logic works but I may be way underestimating the inferential distance. So I’m giving an AI-written expansion below. Workflow: give the same prompt “can you try to write a longer, but still concise version, that explains the logic more?” to 3 SOTA chatbots, then pick the best response, which was from claude-opus-4-6-thinking, by far. The other two mangled some of the concepts pretty badly.)
Let me unpack the logic, because the post is compressed to the point of obscurity:
Step 1: The standard view. Status competition is considered zero-sum. If you and I compete for a promotion, my gain in status/resources roughly equals your loss. Total welfare stays flat. Many people therefore view status-seeking as morally dubious — you’re not creating value, just redistributing it.
Step 2: Introduce “measure.” Some theories in physics and philosophy suggest not all observers carry equal metaphysical weight. Under simulation hypotheses, some beings might run on more computational substrate than others. Under UDASSA (a framework for assigning probabilities to observer-moments), different observers get different “measure” based on algorithmic complexity. “Measure” here means something like: how much does this person’s experience count in the moral ledger of the universe?
Step 3: The punchline. If person A has measure 2 and person B has measure 1, and they’re in a zero-sum status competition where the winner gains +10 utility and the loser gets −10, then:
If A wins: weighted outcome = 2(+10) + 1(-10) = +10
If B wins: weighted outcome = 2(-10) + 1(+10) = −10
The “zero-sum” game now has a clear winner from a cosmic utility perspective. A winning is actually positive-sum when you weight by measure. So if you suspect you have high measure, your moral parliament — the internal coalition of values that guides your decisions — might reasonably let your status-seeking impulses win more often, because those competitions aren’t really zero-sum after all.

Wei Dai 14 Feb 2026 10:38 UTC
9 points
2
in reply to: Jacobson’s comment on: Wei Dai’s Shortform
I don’t regret it, and part of the reason is that I find it hard to find people/opportunities to direct resources to that I can be confident won’t end up doing more harm than good. Reasons:
1. Meta: Well-meaning people often end up making things worse. See Anthropic (and many other examples), and this post.
2. Object-level: It’s really hard to find people who share enough of my views that I can trust their strategy / decision making. For example when MIRI was trying to build FAI I thought they should be pushing for AI pause/stop, and now that they’re pushing for AI pause/stop, I worry they’re focusing too much on AI misalignment (to the exclusion of other similarly concerning AI-related risks) as well as being too confident in misalignment. I think this could cause a backlash in the unlikely (but not vanishingly so) worlds where AI alignment turns out to be relatively easy but we still need to solve other AI x-risks.
(When I did try to direct resources to others in the past, I often regretted it later. I think the overall effect is unclear or even net negative. Seems like it would have to be at least “clearly net positive” to justify investing in Anthropic as “earning-to-give”.)

Wei Dai

Claude is dropping max plans for enterprise (maybe for everyone?)

Eliezer 2018

2018 FB discussion

Me (Wei Dai) 2019