Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
I disagree vibes wise re: “catastrophically screwed up”, but if that’s what you believe, I would suggest looking at the Bell et al. paper and trying to define RL algorithms that tend towards the sort of behavior you think is not catastrophically screwed up.
With multi agent training, it is possible to give everyone the same reward / payoff function, in that case ‘sabotage’ is not really rewarded. For training swarms to solve problems, that’s a fine starting point.
If you know how the RL algorithms work, it is not surprising that RL will make models defect. This is not specific to GRPO, it applies to PPO, REINFORCE, etc. As such it is a “tool for the job” issue. If what you want is approximating CDT self-ratification, use RL. That’s enough for a lot of purposes, like making AI that is good at math.
If you want “make model get high reward as it is trained, even as multiple samples from the model go into the reward, but you only label some as ‘actions’”, you want something pretty different from current RL algorithms, perhaps something more in the MIRI decision theory vicinity. (But also maybe current AIs are safer / more predictable if they do CDT self-ratification than if they do FDT/UDT! Since ‘coordination’ has been named as an alignment concern.)
It seems like we disagree vibes wise but the main thing I’m saying is, this is what RL should do according to theory (like the Bell et al paper), and the theory for the alternative kind of RL that doesn’t do this hasn’t been formalized. It is not a sign of “you screwed up your training algo catastrophically”, it’s what RL theorists would expect, and is not a sign of an implementation error.
“actual dataset”: I was thinking of doing multiple rounds. So after you did a step 3 fine tuning step, run step 1 again, generating samples from the fine-tuned model. This way, if the initial base model is low-perplexity, and step 2 selects for higher perplexity, then the first fine-tune (step 3) will increase the model’s perplexity. What that means is that, in the next round, the step 1 samples will already be higher perplexity, so the round-2 step-3 will be getting a higher loss.
I don’t think the lever analogy is valid. The core thing is that the policy’s reward is influenced by multiple copies of the policy, and only some influences are “gradient descended / RL’d on”. This is a ratification situation (Bell et all) where CDT is favored.
Here’s a next token predictor analogy that might help make sense of ‘loss going up’.
Suppose we start with a base model and train it in the following way:
Take some samples from it (at temperature = 1)
Perform some selection: score them and take the top 10%
Fine-tune the model to predict the selected runs better
Naively, one could reason as follows: “The fine-tuning loss equals the total surprisal of the data. Therefore there is an incentive for the model to move towards outputting predictable text, such as ‘-------------------’. If it does this, then its loss in stage 3 is lower.”
But of course this isn’t what happens. Instead, this procedure will, while inclining it towards producing highly-rated samples, not incline it towards predictable samples.
And that’s a feature, not a bug. When doing “SFT on selection”, you want it to replicate the distribution, not minimize surprisal.
If, for some reason, the base model was initialized towards something low-perplexity, but it happened that the highly-rated samples were on average higher-perplexity, then it would drift towards higher perplexity over time. This would cause an increase in loss over time, as its stage-3 performance gets worse (as stage 3 is now predicting higher-perplexity text).
(I’m not claiming that this is strongly analogous to the twin PD stuff, but it is analogous on the axis on which you’re making the analogy, and I think it argues against the “brain damage” interpretation.)
Hindsight is 20⁄20, but the way I might have predicted this directional change is by thinking about SFT results (e.g. those by Owain Evans like subliminal learning) which suggest that SFT behaves somewhat like “Bayesian updating in parameter space”, and guess that RL is like SFT in this way, so that RL will be sort of like a “Bayesian update parameters on taking behaviors that are RL-stable”, and those are parameters that run something CDT-ish.
Link re: Alex’s claim seems broken?
What would you want to see in a behavioral benchmark? I’d assume it’s obvious that in this twin PD RL setup, agents will tend to defect more often as they are RL’d? See also footnote 1 regarding theory:
Bell et al. (2021) prove that in settings where actions do not screen off policies from rewards, value-based RL agents which explore sufficiently can converge only to policies characteristic of CDT. Oesterheld et al. (2024) conjecture that language models might generalize to behave broadly in accordance with CDT if trained in such settings, and Mallen (2026) has argued that this is indeed likely for reward-seeking models.
I’d register a prediction that Parfit’s Hitchhiker RL, and ‘voting with a minor cost to voting’ RL, would also increase CDT sympathies. (The payoff matrix for ‘voting with a minor cost to voting’ seems much like twin PD, and it seems EDT would vote for similar reasons to FDT?). Based mainly on theoretical reasons such as those indicated in this footnote:
Bell et al. (2021) prove that in settings where actions do not screen off policies from rewards, value-based RL agents which explore sufficiently can converge only to policies characteristic of CDT. Oesterheld et al. (2024) conjecture that language models might generalize to behave broadly in accordance with CDT if trained in such settings, and Mallen (2026) has argued that this is indeed likely for reward-seeking models.
Separately I’d predict that a three-way between CDT / EDT / FDT would also show tendency towards more CDT endorsement when RL’d on twin PD (as indicated by the open-ended favorite decision theory question).
That’s GRPO, a RL algorithm, not the reward. If you used other RL algorithms like PPO, I assume you would get similar results. (GRPO can be justified as approximately unbiased relative to REINFORCE)
Would you consider yourself worried about AI cybersecurity in the near term?
If so, do you have an approximate answer for his question 2? (Even a very vague estimate such as: Percent increase in costs of cybersecurity attacks and/or hardening over the next 3 years)
Your current numerical estimate of what those costs might end up being, of course to be tested against what actually happens over time. Obviously, you can do this for a few different regulatory/safety scenarios. This is one simple way to prove yourself largely correct, albeit with a lag.
If 1-yes 2-no, do you think this reflects badly on the epistemology of people raising alarm about AI cybersecurity while giving no numerical estimates?
Ah, that clarifies things. (To be clear I’m pro science & conceptual progress, am not anti capabilities, and am skeptical of the alignment / capabilities distinction as presently theorized. I agree regarding philosophical progress and scientific understanding changing concepts.)
I’ll leave aside “did alignment leaders act net negatively with respect to their original goals?” and focus on what the harms were and how they happened. As you say, Sam Altman and other AI leaders did a bunch of AI capabilities advancement, and in some ways benefitted from the alignment community, with respect to recruiting, funding, and narrative.
EA is mostly distinctive here in being a group of people who significantly disagreed with MIRI while still doing a bunch of interpretive labor and also having power, wealth, and ambition. (I blame utilitarianism for FTX but not for OpenAI.)
Basic factors that may have contributed:
a) Disagreement about crucial considerations
b) Interest in being personally important / powerful / rich
c) Inevitability intuitions as you discuss. (I’m undecided about how correct / useful such intuitions were. While there is a prima facie case against “do bad because someone would otherwise do bigger bad later”, the specifics matter, the world is big and low-trust, empirical/historical updates are justified rather than thinking you control the past, international conflict is a thing, security mindset is inherently low-trust, etc)
d) Resenting MIRI for having good arguments while not being sufficiently prestigious, deeply correct, etc
My high-level vibes-based model is that MIRI has made a lot of claims that are high in “big if true”, “hard to think about well”, “morally demanding if true”, “high theoretical uncertainty”, “you should give us money if this is true”, etc, and that such claims led to a high degree of engagement by people trying to be (or at least, seriously trying to appear) intellectually honest, but that high amounts of disagreement, emotional difficulties, resentment, conflict, and so on, are expected on this reference class, and that this leads to legible agendas being subverted. Some of this is “social dark matter” that MIRI!2010 would not have expected, and they would have strategically acted differently if they knew about it.
Low hanging fruit: There used to be a bot (“RemindMeOfThis” or similar) people could tag to get a reminder in some time frame. For some reason I don’t see a currently working version of this. I assume a contributing factor is that the X API is now expensive.
While it doesn’t globally “work better”, yes CDT+SIA is guaranteed to have solutions in multi player absent minded games, and UDT isn’t.
There are also EDT+FNC (aka EDT+GDH) equilibrium concepts, which do not always exist in multiplayer games (see “Imperfect-Recall Games: Equilibrium Concepts and Their Complexity”, Lemma 21).
I do think “multi player equilibria always exist” is a pretty desirable property for an equilibrium concept, although the lack of ex ante optimality is an issue. (Some of my previous writing on this: “Buridan’s ass in coordination games”, “Reducing collective rationality to individual optimization in common-payoff games using MCMC”, both making use of shared randomness, a workaround for Wichardt 2008 style counterexamples; as an offhand idea, it might be interesting to examine how logical inductor EDT handles such cases.)
Absent minded games can provide “easier” models of Newcomblike games; quoting a previous post,
Memoryless Cartesian environments can be used to define many familier decision problems (for example, the absent-minded driver problem, Newcomb’s problem with opaque or transparent boxes (assuming Omega runs a copy of the agent to make its prediction), counterfactual mugging (also assuming Omega simulates the agent)). Translating a decision problem to a memoryless Cartesian environment obviously requires making some Cartesian assumptions/decisions, though; in the case of Newcomb’s problem, we have to isolate Omega’s simulation of the agent as a copy of the agent.
Which is part of why I’m focusing on them (as they seem less general / easier to analyze).
My concern is that the alignment community had a plan to make good outcomes more likely (differentially advancing alignment over capabilities) but has mostly pushed the world in the opposite direction, while some parts of it gained a lot of power by doing so.
Not sure how you’re assessing the counterfactual. It’s easy to say “here, alignment research accelerated capabilities!” but what does the ratio look like if there had been no public attempt at doing alignment research? (Or if ratio is not the right thing, how else to assess?)
I also think the strategic focus on “capabilities bad!” is in tension with the praise for scientific and conceptual progress:
More generally, one touchstone I’ll be referring to throughout this sequence is the idea that scientific progress proceeds by developing insightful new concepts
Unfortunately, most alignment research is no longer even aiming towards the kind of scientific progress I describe above.
So the kind of conceptual thinking that I’m praising in the rationalist community is what I’d call the generative part of science
Since… of course, if this were true, then alignment research (including agent foundations) would accelerate capabilities, right? Understanding how things work helps to make things work better? Obviously?
Yes, I realize applause lights around here are “capabilities bad” and “science good” and “conceptual progress good”, but if you are urging more careful ethical and strategic reflection on the part of others, then perhaps consider examining this more carefully.
You can say “well yes of course but it’s worth the tradeoff” however the way you’re describing not-obviously-applicable conceptual progress in glowing terms doesn’t seem concordant with making such tradeoffs. And again I don’t know what underlying model you are using to assess such tradeoffs.
This is not merely a theoretical concern, in that MIRI’s research progress drastically slowed at around the time they became more secretive, security conscious, and strategically focused (~2017).
It’s useful to contrast prestige-orientation with Eliezer’s alternative recruitment strategy—writing Harry Potter and the Methods of Rationality—which was closer to a prestige-minimizing move, yet which was much more successful in recruiting people who could think clearly about alignment.
Not convinced. Check your own sources, the papers/posts you give as especially good progress. Many of these people are educated and would be more likely brought in by the things you put in the “prestige” bucket (like Superintelligence) than the “fanfic” bucket.
(I do appreciate you are giving a history of the field/community and trying to examine ethical issues! Focusing on criticisms here as I have more to say there.)
Primarily 2, with the extra bit where we can be assured there’s a CDT+SIA fixed point. For the purpose of anthropic probabilities, that’s a desirable condition. We don’t need probabilities to force optimality but we would rather they be compatible with optimality, and preferably that they help with the computation.
The other part is that UDT is not really computationally tractable and so figuring out how to optimize it is useful, which is more (1) territory. For this the computational complexity paper is relevant because it shows CDT+SIA (which they call CDT+GT) gives KKT conditions. So we can restrict to CDT+GT points when searching for ex ante optimal policies.
One idea also is to relax to “list CDT+GT policies but without the inequalities and allowing complex numbers” which is now algebraic geometry territory (solving complex polynomials), and then filter for inequalities, real numbers, and global optimality afterwards.
From reading through the Sleeping Beauty literature, I learned about the Wichardt 2008 counterexample for multi-player ex ante optimality, which is relevant to UDT (and says why we shouldn’t expect nice things for UDT in general).
To give a brief summary, suppose Alice has 2 copies, who have the same source code and who can randomize independently. There are 2 coffee shops that the copies can decide to go to without communicating. They would really like (+5) to meet at the same coffee shop. Also, Bob hates Alice, his utility function is hers negated. Alice gets −1 utility (and Bob +1) if Bob goes to the coffee shop where both Alice-copies go. (Basically Alice is playing a “coordination” variant of absent-minded driver with herself, while also playing matching pennies with Bob)
The problem is that they can’t both select ex ante optimal policies in the Nash equilibrium sense. Alice needs to either 100% go to coffee shop A or B to be ex ante optimal; suppose it’s A. But then Bob needs to 100% go to coffee shop A to be ex ante optimal. Now Alice isn’t ex ante optimal anymore because it would have been more optimal for both her copies to go to coffee shop B.
If we treat it as a 3 player game then we have a Nash equilibrium (both Alices go to A, Bob goes to A). This is CDT+SIA optimal on everyone’s part (the SIA part doesn’t matter for these purposes). Since neither Alice-instance can unilaterally switch and expect things to (causally) work out well.
Given this impossibility result it’s not clear what UDT-like decision theories should do in these sorts of situations. Since it means not everyone can be ex ante optimal and maybe we need weaker conditions too.
So the whole direction of justifying probabilities by decisions is probably a dead end.
CDT+SSSA+SIA gives KKT conditions, covers all UDT policies in absent-minded games, is ex ante optimal in additive games.… A bit more discussion in other post. The part where number of copies is affected does not prevent CDT+SSSA+SIA from being well-defined (in the sense of having self-ratifying probability / policy combinations), or from giving ex ante utility stationarity conditions. It just complicates the scenario, which is why I didn’t focus on it in this post.
Alternatively, if you view probabilities as “reality fluid” that determines what to expect in the next moment, that’s an important question too, but it’s not clear why Dutch books help solve it.
This seems more in metaphysics territory. I take the physicalist baseline to be that one should assign probabilities to centered worlds and that’s all. Maybe there is other stuff to assign probabilities to (like qualia, idk). I think before assigning numbers a thing to do is to figure out what is the metaphysical status (if any) of the entity being assigned probabilities. Since it does not seem clear to me that there is a there there. (Whereas with things like the Sleeping Beauty problem or Doomsday Argument it seems there is plenty of disagreement to be had about rational probabilities on centered worlds, even without getting into any metaphysics on top of centered worlds)
Just finished up a post on probabilities over centered worlds.
SSSA+SIA does not assign consistent probabilities over full personal histories. (It assigns probabilites over centered worlds and some amount of personal history probability can be derived from that, yet won’t satisfy assumptions of OP for personal probabilities, e.g. won’t consider
In standard Sleeping Beauty there is no memory loss from Sunday → Monday. This is part of David Lewis’s argument for single-halfing (using Martingale assumption). So violating Martingale even when your mind hasn’t been messed with in the intervening time is standard thirding.
With duplicate Sleeping Beauty we have analogous Dutch book arguments for thirding. If you assign
(One can appeal to utility function re: shared accounts, however, if it’s a shared account for an altruistic fund, which all copies of Beauty care about equally, we still have a problem)
Someone can hold “I don’t buy Dutch book arguments at all, I think Martingale can be supported on other assumptions”, however historically, diachronic Dutch book arguments have been a significant argument for Martingale.
I agree SIA is non-martingale (common thirder property). I think one thing you are saying is that if we agree with “simple Bayes” we can’t have probabilities over personal histories, and a big motivation for SIA is simple Bayes, so SIA-ers will tend to not assign consistent probabilities over personal histories; I’d agree there.
I feel that it probably has timeline inconsistencies, such as assigning a different size to the universe at different moments, without any observations to update on.
This is standard thirding. In standard Sleeping Beauty, between Sunday → Monday, Beauty gets no non-trivial observations. Yet her epistemic state changes, in the following sense: Usually, you should move your ‘current time’ beliefs forward according to a clock (assuming you have a good internal clock). However, from Sunday → Monday, Beauty should assign some probability to “it is Tuesday” which is not updating according to an internal clock. (Actually, halfers agree here, that upon awakening, she should consider it plausible that it’s Tuesday.). Thirders correspondingly assign more probability to Tails in a non-Martingale fashion (and this is not so counter-intuitive if you account for the internal clock issue). This is a probability for an objective event that happened in the past predictably changing. It violates Martingale, yet Dutch book arguments suggest that violating Martingale here is necessary to avoid sure losses (assuming CDT, which is the common background Dutch book assumption).
Analogous case for universe size: Number of times Beauty wakes up depends on how scientists measure size of universe on Monday (on Sunday no one knows what the size is).
“All the best agents are FDTs/EDTs, but CDT is what works” sort of situation. (That is, I expect some of the capability / EDT correlation to be persona selection, but that RL on problems that distinguish CDT from EDT/FDT will push towards CDT behaviors, and have some indirect influence on persona that way, as OP suggests.)