Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
I am not even convinced the tickle defense is relevant because I’m not convinced PPO is doing EDT rather than CDT. Is there an example of a game where you think PPO does something non-CDT (in particular, in an EDT direction)?
I’m not very familiar with RLAIF but I do think the usual method is to have the critic not be much dumber than the agent. Regardless when I am saying PPO does CDT I am not talking about RLAIF, I am talking about reinforcement learning from the game’s payoff matrix.
I’m currently excited about agent foundations precisely as a strategy for aligning otherwise-prosaic neural-network-based AGIs
I’m guessing you’re going to say more specific stuff on this in later posts, but I want to plug some examples of research that seem productive/scientific in the “agent foundations + neural nets” vicinity:
Decision theory, imperfect recall games, RL
In memoryless Cartesian environments, every UDT policy is a CDT+SIA policy
Can de se choice be ex ante reasonable in games of imperfect recall? A complete analysis
The Computational Complexity of Single-Player Imperfect-Recall Games
Kimi likes causal decision theory more after RL in twin prisoner’s dilemmas
(shoutout to Caspar Oesterheld who keeps appearing on these papers)
LLMs and VNM/Bayes
Your claim is that PPO creates EDT. I don’t think it does, I think it creates CDT. I think what you are saying is: PPO creates EDT, but then EDT does a tickle defense and then acts like a CDT. I’m not exactly convinced but this might be a terminological disagreement? It seems like we agree that PPO causes the actions that we would expect from CDT, and there’s a separate question of whether “EDT + tickle defense” is the right way to reason about this, where it seems we disagree, but it’s also vaguer / more terminological?
I think one way to get EDT is to use evolutionary algorithms. If we create some random weights and then select ones that get high scores in twin PDs (against agents with same weights), then, depending on the numbers, the cooperators might get selected. However my impression is that all RL algorithms in current use are more CDT-ish?
Linked paper seems like a RLAIF situation rather than “traditional RL” or RLVR. I suppose the idea is that, for example, an AI evaluator could have high expectations about an AI actor who has one-boxed, and that if those expectations are used as the actor’s reward, then it will one-box.
PPO creates EDT though.
Disagree. I’m going to focus on REINFORCE since PPO mostly differs in the importance sampling / clipping.
We set up a Newcomblike situation as follows. Let
First with REINFORCE we suppose N is huge. What happens is that we take many runs with the same weights. Since
This was with high N. With low N it’s a little harder to reason about, but it is still true that, in expectation, the gradient is towards defection. (Since in expectation, the low-N update is the same as the high-N update, it’s just we get less variance with high N).
You could try making the case that PPO differs from REINFORCE here but that shouldn’t be the default hypothesis.
If I understand the basic idea correctly, it’s that a convex set can be represented by its maxima for all linear functions on the vector space containing that set, and under these assumptions, that amounts to “the set of achievable outcome distributions can be represented by its maximal attainable utility for all utility functions”.
I’m curious what changes when looking at games of imperfect recall (general background discussed in this section; see also this paper for a connection between CDT+GT and KKT conditions).
For example, in the absent-minded driver problem, there are 2 intersections, and 3 different destinations; if you turn at the first, you get to A; if you go straight on first and turn on second, you get to B; if you go straight twice, you get C. Your policy is your probability of turning (since due to amnesia, there is no way to distinguish between the intersections). The set of achievable outcome distributions is not convex, since 100% A and 100% C are achievable, but 50% A, 50% C is not achievable. So we can’t describe the feasible region with utility attainment, as in the convex case.
Making a ‘finite number of time steps’ assumption, we can describe the outcome distribution as a polynomial of the policy probabilities; the policy space can be described as a product of simplices. When running a product of simplices through a polynomial, we get a non convex shape. To apply linear algebra, we could consider representing a degree-d polynomial on a vector space
I’m skeptical of the ‘misalignment’ story here. I’ll present an alternative account.
Gaia, the primitive Earthly aspect of Gnon, rules the hunter-gatherer era. Hunter-gatherers must align with Gaia by hunting and gathering enough food to eat. Without technological advancement, their populations are limited by ecology, regardless of their preferences. Malthusian limits are harsh.
Agriculture initially makes it possible to eat less laboriously. Gaia can be supplicated with fewer labor-hours per person-year. The Malthusian limit is much less harsh. Population increases, until approaching the agricultural Malthusian limit.
Uruk intermediates between Gaia and agricultural humans. Gaia’s rules still apply: if you don’t eat, you die. Uruk will help supplicate Gaia, but makes demands of you, such as producing grain, and being patient. Uruk’s demands are more computationally tractable than Gaia’s demands. By simplifying “if you don’t eat, you die” into “if you don’t farm enough, things will be bad for you”, Uruk reduces the cognitive burdens imposed on humans by Gaia (at least until the next Malthusian limit, at which point, more technologies such as writing have been developed, and efficiency-increasing technologies are on the way).
Uruk is imperfectly aligned with Gaia. Producing lots of grain is very helpful for inclusive fitness, but doesn’t account for aspects such as nutrition. You could compare this to a mesa-optimizer, but it’s perhaps closer to a strong set of heuristics.
The rulers and priests maintain Uruk, including by handling parts of the Uruk-Gaia interface which are less legible. This is cognitively costly, like handling hardware-software alignment. Also, the Uruk system favors some humans over others. There is not really a unified “Humanity” who is benefitted or harmed by Uruk. Uruk creates winners and losers.
How would we even tell what is good for “human values”? We can try inferring human values from two main sources: stated and revealed preferences. By stated preferences, many humans praised the Uruk system. By revealed preferences, many humans acted to maintain the Uruk system. These are not perfect indicators, since, for example, stated preferences can be affected by direct threats from others. But the bar for claiming false consciousness should be fairly high. Since both optimistic and pessimistic literature exists, one cannot easily derive that only the pessimistic literature expresses human values.
Also, statistics about average welfare are insufficient to establish misalignment, for a few reasons. First, because not everyone is an average utilitarian; some are more total utilitarian, or non-utilitarian. Second, because there are winners and losers, and so “<X technological/cultural advancement> is aligned with some humans but not others” should be considered. Third, because average welfare statistics don’t say much about other values, including aesthetic values, and values about the future (the post-Uruk period).
What is easier to establish is that a proxy objective exists. Proxy objectives can be pretty great. For example, proxy objectives have made it possible to simplify the task of eating to things like “making money” and “buying items from grocery stores”. This decomposition is imperfect. Less-intermediated Gaia optimization would generate other ideas, like dumpster diving, theft, garden agriculture, hunting local wildlife, and so on. But even if some of these ideas are, in principle, more efficient ways of attaining inclusive fitness, they are less cognitively tractable than conventional ways of eating. Also, some of these methods would increase costs for others and/or fail to support a high population.
If you optimize for a proxy objective in place of a Gaian objective, then you pay the cost in the fitness arbitrage differential, e.g. in needing advice from priests. And if you optimize for a proxy objective in place of direct subjective expected utility maximization, then you pay the approximation error of expected utility. But proxy objectives have benefits for bounded rationality, since they can be more tractably optimized, and can more tractably be coordinated on.
(Lessons for AI: (a) proxy objectives can increase intelligence by relaxing bounded-rationality constraints, e.g. in RLVR, while in some cases reducing replicator fitness, (b) assessing the impact of AI, especially near-term AI, on ‘human values’ is complicated by, among other things, that there are winners and losers, and that people’s stated and revealed preferences vary, (c) effective coordination between humans involves proxy objectives, therefore cannot tractably be on Gaian or subjective-expected-utility objectives)
You could set up twin PD so it’s very similar. What you do is, if p is the cooperation probability (determined by the weights, which have been perturbed), you simulate
It could be that p is now different between the different weights in the group. There could be a correlation between one-boxing and reward within the group because of this (depending on exact numbers). I’m not familiar with combining GRPO with weight randomization. My guess is that it could reinforce one-boxing as runs that one-box have higher expected advantage. (This is perhaps somewhat easier to see by looking at REINFORCE and then noting that GRPO is similar.)
Yes two boxing is what it converges to under standard RL such as REINFORCE or GRPO.
The idea is if the probability is ‘p’ of one boxing and ‘x’ is its action (1 for 1 box, 0 for 2 box) what you do is reward it with 10p—x. Note p is a deterministic function of weights, whereas x is sampled randomly.
Within a grpo group, p is the same for everyone, it’s just x that changes in the group. Since the coefficient of x in reward is negative, the reinforcement is towards ‘not x’. That gradient update reduces p over time.
This is not unique to grpo, it’s just easier to explain there.
(To make analysis simpler we can assume p is clipped to be in the range [0.001, 0.999] so exploration happens)
“All the best agents are FDTs/EDTs, but CDT is what works” sort of situation. (That is, I expect some of the capability / EDT correlation to be persona selection, but that RL on problems that distinguish CDT from EDT/FDT will push towards CDT behaviors, and have some indirect influence on persona that way, as OP suggests.)
I disagree vibes wise re: “catastrophically screwed up”, but if that’s what you believe, I would suggest looking at the Bell et al. paper and trying to define RL algorithms that tend towards the sort of behavior you think is not catastrophically screwed up.
With multi agent training, it is possible to give everyone the same reward / payoff function, in that case ‘sabotage’ is not really rewarded. For training swarms to solve problems, that’s a fine starting point.
If you know how the RL algorithms work, it is not surprising that RL will make models defect. This is not specific to GRPO, it applies to PPO, REINFORCE, etc. As such it is a “tool for the job” issue. If what you want is approximating CDT self-ratification, use RL. That’s enough for a lot of purposes, like making AI that is good at math.
If you want “make model get high reward as it is trained, even as multiple samples from the model go into the reward, but you only label some as ‘actions’”, you want something pretty different from current RL algorithms, perhaps something more in the MIRI decision theory vicinity. (But also maybe current AIs are safer / more predictable if they do CDT self-ratification than if they do FDT/UDT! Since ‘coordination’ has been named as an alignment concern.)
It seems like we disagree vibes wise but the main thing I’m saying is, this is what RL should do according to theory (like the Bell et al paper), and the theory for the alternative kind of RL that doesn’t do this hasn’t been formalized. It is not a sign of “you screwed up your training algo catastrophically”, it’s what RL theorists would expect, and is not a sign of an implementation error.
“actual dataset”: I was thinking of doing multiple rounds. So after you did a step 3 fine tuning step, run step 1 again, generating samples from the fine-tuned model. This way, if the initial base model is low-perplexity, and step 2 selects for higher perplexity, then the first fine-tune (step 3) will increase the model’s perplexity. What that means is that, in the next round, the step 1 samples will already be higher perplexity, so the round-2 step-3 will be getting a higher loss.
I don’t think the lever analogy is valid. The core thing is that the policy’s reward is influenced by multiple copies of the policy, and only some influences are “gradient descended / RL’d on”. This is a ratification situation (Bell et all) where CDT is favored.
Here’s a next token predictor analogy that might help make sense of ‘loss going up’.
Suppose we start with a base model and train it in the following way:
Take some samples from it (at temperature = 1)
Perform some selection: score them and take the top 10%
Fine-tune the model to predict the selected runs better
Naively, one could reason as follows: “The fine-tuning loss equals the total surprisal of the data. Therefore there is an incentive for the model to move towards outputting predictable text, such as ‘-------------------’. If it does this, then its loss in stage 3 is lower.”
But of course this isn’t what happens. Instead, this procedure will, while inclining it towards producing highly-rated samples, not incline it towards predictable samples.
And that’s a feature, not a bug. When doing “SFT on selection”, you want it to replicate the distribution, not minimize surprisal.
If, for some reason, the base model was initialized towards something low-perplexity, but it happened that the highly-rated samples were on average higher-perplexity, then it would drift towards higher perplexity over time. This would cause an increase in loss over time, as its stage-3 performance gets worse (as stage 3 is now predicting higher-perplexity text).
(I’m not claiming that this is strongly analogous to the twin PD stuff, but it is analogous on the axis on which you’re making the analogy, and I think it argues against the “brain damage” interpretation.)
Hindsight is 20⁄20, but the way I might have predicted this directional change is by thinking about SFT results (e.g. those by Owain Evans like subliminal learning) which suggest that SFT behaves somewhat like “Bayesian updating in parameter space”, and guess that RL is like SFT in this way, so that RL will be sort of like a “Bayesian update parameters on taking behaviors that are RL-stable”, and those are parameters that run something CDT-ish.
Link re: Alex’s claim seems broken?
What would you want to see in a behavioral benchmark? I’d assume it’s obvious that in this twin PD RL setup, agents will tend to defect more often as they are RL’d? See also footnote 1 regarding theory:
Bell et al. (2021) prove that in settings where actions do not screen off policies from rewards, value-based RL agents which explore sufficiently can converge only to policies characteristic of CDT. Oesterheld et al. (2024) conjecture that language models might generalize to behave broadly in accordance with CDT if trained in such settings, and Mallen (2026) has argued that this is indeed likely for reward-seeking models.
I’d register a prediction that Parfit’s Hitchhiker RL, and ‘voting with a minor cost to voting’ RL, would also increase CDT sympathies. (The payoff matrix for ‘voting with a minor cost to voting’ seems much like twin PD, and it seems EDT would vote for similar reasons to FDT?). Based mainly on theoretical reasons such as those indicated in this footnote:
Bell et al. (2021) prove that in settings where actions do not screen off policies from rewards, value-based RL agents which explore sufficiently can converge only to policies characteristic of CDT. Oesterheld et al. (2024) conjecture that language models might generalize to behave broadly in accordance with CDT if trained in such settings, and Mallen (2026) has argued that this is indeed likely for reward-seeking models.
Separately I’d predict that a three-way between CDT / EDT / FDT would also show tendency towards more CDT endorsement when RL’d on twin PD (as indicated by the open-ended favorite decision theory question).
That’s GRPO, a RL algorithm, not the reward. If you used other RL algorithms like PPO, I assume you would get similar results. (GRPO can be justified as approximately unbiased relative to REINFORCE)
Would you consider yourself worried about AI cybersecurity in the near term?
If so, do you have an approximate answer for his question 2? (Even a very vague estimate such as: Percent increase in costs of cybersecurity attacks and/or hardening over the next 3 years)
Your current numerical estimate of what those costs might end up being, of course to be tested against what actually happens over time. Obviously, you can do this for a few different regulatory/safety scenarios. This is one simple way to prove yourself largely correct, albeit with a lag.
If 1-yes 2-no, do you think this reflects badly on the epistemology of people raising alarm about AI cybersecurity while giving no numerical estimates?
Ok, we could set up the game so that the input string is a uniformly random element of the finite set
It might depend what the network actually is, but in the simple case,
Again we consider REINFORCE with a huge N. We get an approximate uniform sampling from
In conclusion, under this setup, REINFORCE is going to act like CDT. There is some possible variation in how to lay out the network (I made the simplifying assumption here), though I’m guessing for a sufficiently big network, able to encode “practically all” functions