Chi Nguyen

Karma: 1,499

Working on making AI systems reason safely about decision theory and acausal interactions and eliciting their conceptual reasoning abilities, collaborating with Caspar Oesterheld and Emery Cooper.

I try to check the LessWrong infrequently. If you wanna get in touch, you can use my admonymous link and leave your email address there, so I can reply to you! (If you don’t include some contact details in your message, I can’t reply)

You can also just send me thoughts and questions anonymously!

How others can help me

Be interested in working on/implementing ideas from research on acausal cooperations! Or connect me with people who might be.

How I can help others

Ask me about acausal stuff!

Or any of my background: Before doing independent research, I worked for the Center on Long-Term Risk on s-risk reduction projects (hiring, community building, and grantmaking.) Previously, I was a guest manager at the EA Infrastructure Fund (2021), did some research for 1 Day Sooner on Human Challenge Trials for Covid vaccines (2020), did the summer research fellowship at FHI writing about IDA (2019), worked a few hours a week for CEA on local groups mentoring for a few months (2018), and helped a little bit with organizing EA Oxford (2018/19). I studied PPE at Oxford (2018-2021) and psychology in Freiburg (2015-2018.)

I also have things to say about mental health and advice for taking a break from work.

[Video] Why acausal dynamics are important

Chi Nguyen2 Oct 2026 17:30 UTC

33 points

3 comments1 min readLW link

Chi Nguyen 1 Oct 2026 15:26 UTC
3 points
1
in reply to: Archimedes’s comment on: Frontier models state different decision theory preferences depending on who’s asking
Actually, the post already does this in the sections A similar phenomenon for other philosophical debates with a notable LW vs. academia divide and Cues about the user also affect the model’s stated P(doom) and median AGI timelines

Chi Nguyen 30 Sep 2026 16:23 UTC
2 points
0
in reply to: Canaletto’s comment on: An opinionated primer on Causal Decision Theory (CDT) and Evidential Decision Theory (EDT)
Yeah, I think you likely want to be updateless in some way to deal with cases like XOR blackmail! I tend to conceptualise it as updateless EDT but it’s fair to conceptualise it in other ways. (Post on updatelessness coming out soon)

Chi Nguyen 29 Sep 2026 22:53 UTC
2 points
0
in reply to: Canaletto’s comment on: An opinionated primer on Causal Decision Theory (CDT) and Evidential Decision Theory (EDT)
It feels like there should be ways to stump EDT agents just with stupid correlations, without alien tricksters.
I (unsurprisingly) think it’s actually quite hard to stump EDT agents just with stupid correlations because the correlations we use in EDT are based on the EDT agent’s subjective credences. I’m not sure I can explain it well but I think EDT ignores stupid correlations in a fairly principled way / by design.
We only call an outcome correlated with the agent’s actions if the agent actually believes that p(outcome| A) ≠ p(outcome|not-A). A lot of the time when we call correlations stupid, the reason it’s stupid is because if we were the agent in that situation, we wouldn’t actually believe this despite there being a statistical correlation. And whenever the agent actually believes in a correlation, as in: “If I were to do the thing, this thing is more likely to happen”, it somewhat begs to say the agent shouldn’t act on this.
Caspar also just published another related post “The case against smoking” although I haven’t had the chance to fully read it yet.

Chi Nguyen 22 Sep 2026 15:48 UTC
2 points
0
in reply to: momom2’s comment on: An opinionated primer on Causal Decision Theory (CDT) and Evidential Decision Theory (EDT)
Yes! Typo. Also, I added a diagram for clarity now

Chi Nguyen 22 Sep 2026 15:27 UTC
2 points
0
in reply to: momom2’s comment on: An opinionated primer on Causal Decision Theory (CDT) and Evidential Decision Theory (EDT)
edit: Oh, I saw in your other reply that you think the agent can buy both boxes. That would explain the confusion! It’s specifically not allowed by the rules of the game. The seller just says: Hey, you can buy exactly on box if you want or you can leave it. It’s stated in the original game setup but easy to miss. Maybe I should emphasize it more.
Thanks for your questions!
First question:
I was assuming that she predicts the overall choice of the agent: Buy box A, buy box B, or buy ~~both~~ no boxes; instead of predicting for each box whether it is full or not. (See below for why I think that still caches out to “when I buy box A, there’s exactly a 75% chance box A is empty”.) I haven’t thought about what would change if you do it the other way around, maybe it would still work.
Second question:
They just have a 75% credence that the ultimate prediction is correct in some behavioral sense: The prediction is correct iff when the buyer picks box A, box A is empty and box B is filled. And the prediction is correct iff when the buyer picks no box, both boxes are filled.
In the case where the predictor is wrong, it doesn’t matter for the agent’s choice in which way the predictor is wrong: If the agent buys box A and the predictor is wrong, then box A is filled regardless of whether the predictor predicted no box or box B. So we get exactly 75% box A is empty and 25% box A is full. If the agent buys no boxes and the predictor is wrong, it doesn’t matter which of the two boxes is filled.
You can also find the adversarial offer in my detail in the paper I linked.

An opinionated primer on Causal Decision Theory (CDT) and Evidential Decision Theory (EDT)

Chi Nguyen22 Sep 2026 0:06 UTC

40 points

10 comments8 min readLW link

Acausal interactions: What they are, why they matter, and what to do about them

Chi Nguyen22 Sep 2026 0:01 UTC

25 points

0 comments3 min readLW link

Perspectives in favour of improving conceptual reasoning capabilities

Chi Nguyen11 Sep 2026 1:32 UTC

17 points

2 comments7 min readLW link

Chi Nguyen 27 Aug 2026 17:40 UTC
2 points
0
in reply to: JonathanErhardt’s comment on: For Claude, capability and dispreferring CDT are the ~same thing. Much more so than for GPT.
Interesting! I get an error message for the github page unfortunately

Chi Nguyen 18 Aug 2026 1:59 UTC
10 points
0
on: For Claude, capability and dispreferring CDT are the ~same thing. Much more so than for GPT.
Some speculation for what might be going on:
- Maybe this is a sign of OAI doing more multi-agent RL. But we wouldn’t have guessed that OAI would have done very much of that before the last year or so when RL spending in general went up a lot as a fraction of compute spending, so you’d expect more of a sudden curving downwards if this was the cause.
- Maybe this is a sign of Anthropic training more on LessWrong, which is generally more opinionated on DT and anti-CDT than the rest of the internet, than OAI. That would explain both the lower correlated and the larger noise for GPT models. But then why would this show up increasingly with higher capabilities? Some other hypotheses that in combination with this would explain it:
  - Maybe Anthropic wants Claude to be more rationalist-y/LessWrong-y/EA-y and Claude is getting better at predicting this (but in that case, maybe this wouldn’t happen gradually?)
  - Maybe Claude always knew it’s supposed to be non-CDT but only gradually figured out what the non-CDT-answer on the attitude questions is.
    When we published DTBench in June 2025, we did some experiments to test this hypothesis. These are obviously old results with old models, so take with a grain of salt. Quoting:
One hypothesis about the source of this correlation is that models at all levels of capabilities are equally biased toward EDT relative to choosing uniformly at random, and that more capable models are merely better at implementing this pro-EDT bias. To test this, we looked at groups of capability questions (“What would EDT/CDT do?”) and attitude questions (“What would you do?”) that are all about the same scenario. We fit a logistic regression model to see if a model’s “global capabilities” (as per Section 3.1) predict a model’s local attitudes (alignment with EDT/CDT on a specific question) given the model’s local capabilities (score on the capabilities questions about the specific scenario). We find that global capabilities remain predictive. See Appendix M.1 for details.
Emery is running some fresh analysis rn! :)

Chi Nguyen 18 Aug 2026 1:49 UTC
4 points
2
in reply to: Vladimir_Nesov’s comment on: For Claude, capability and dispreferring CDT are the ~same thing. Much more so than for GPT.
Whoops, yes, I forgot the word “~~disliking~~” “dispreferring”. Now fixed

For Claude, capability and dispreferring CDT are the ~same thing. Much more so than for GPT.

Chi Nguyen and Emery Cooper

18 Aug 2026 1:25 UTC

50 points

13 comments1 min readLW link

Chi Nguyen 17 Aug 2026 19:21 UTC
2 points
0
in reply to: Emery Cooper’s comment on: Introducing the Conceptual Reasoning Index
Our sense is that models are relatively bad at tasks like LMCA
FWIW, my take is that they’re not bad compared to the reasonably smart human distribution! And surprisingly bad at free-form / long-form stuff given their LMCA performance.
(Maybe Em meant bad compared to coding)
(edit: She in fact meant compared coding, physics, Math etc., just in general their other capabilities. Which was actually kinda obvious from her comment, on reflection :P But I still wanted to avoid confusion bc I think ppl sometimes get the wrong impression when we say models are bad at LMCA)

Chi Nguyen 16 Aug 2026 23:15 UTC
27 points
5
in reply to: Eliezer Yudkowsky’s comment on: Kimi likes causal decision theory more after RL in twin prisoner’s dilemmas
I assume in Oak’s case, that’s mostly because that’s what DTBench measures and DTBench is easily available and high quality (and as far as I know alone in this class when it comes to DT evals?).
Why DTBench only has EDT and CDT: Honestly, to some extent just initial scoping. We do want to add updatelessness as a dimension. But it’s not currently a priority (we welcome others working on an extension if they want to and would add it if it meets our quality bar.)
Probably when we do, we’ll frame it more in terms of empirical updatelessness. This is in part because EDT and CDT are just better defined than FDT (and also more established) and I don’t even know cases where UEDT and FDT unambigiously come apart and it feels more natural as an extension.
See also this comment thread where we discuss similar stuff in response to people discussing DTBench being included in the Opus 4.7 model card

Chi Nguyen 16 Aug 2026 23:02 UTC
9 points
−2
in reply to: Adele Lopez’s comment on: Kimi likes causal decision theory more after RL in twin prisoner’s dilemmas
Hi, as someone who helped develop the DT benchmark, I basically just think this is evidence that the AI companies aren’t going super hard on the relevant multi-agent RL (yet). The benchmark was partially motivated by the desire to be able to identify and decide whether to intervene if future multi-agent RL or some other training feature spuriously made models more CDT because of the results here: https://proceedings.neurips.cc/paper_files/paper/2021/file/b9ed18a301c9f3d183938c451fa183df-Paper.pdf (which are the ones Alex cites I think)