Homepage: 0ak.hu
oakhu
Karma: 394
Directionally, but an extremely weak effect (hard to distinguish from sampling noise, or lightly frying the model just by training in weird environments) as judged by the capabilities portion of DTBench. But it’s possible that directly training for CDT endorsement (rather than generalizing from training for defection) could have stronger effects.
Good question! It does make those indistinguishable, and that’s what you want for standard RL. In terms of raw score, the trajectories which defect still outperform the trajectories which cooperate in expectation (there’s an example calculation in footnote 7), so you’re still moving towards defection overall on average. But if you just reinforce according to the raw score, then intuitively you’re just adding some more movement towards (or away from) all of the trajectories that happened to get sampled, which introduces a lot of noise (including in all sorts of random directions) without changing the average effect. (It would even be possible, though extremely unlikely, for an individual training step to move the policy towards cooperating more, for example if every single trajectory cooperated; but on average you’d still move towards defecting.)
For intuition, it might help to recall that you can’t increase both P(A) and P(¬A); if you fiddle with things to make A one percentage point more likely, and also fiddle with things to make ¬A one percentage point more likely (or fiddle with things to double both of their log odds), this cancels out in terms of the effect on P(A), though you’ve still ended up perturbing things in some random way.
So the “RL trains for defection in prisoner’s dilemmas” part of the story isn’t too surprising, or sensitive to these algorithmic choices (for instance, it’s been studied in Bell et al. 2021); the main thing we found interesting here was the generalization to endorsing CDT more.
The chess analogy is just a way to see that how the policy’s self-play performance changes can come apart from how the policy’s performance against a fixed opponent changes; here, the former goes down over training, but the latter goes up (intuitively, because the RL algorithm cares about the latter); you could make the chess game nominally positive sum by counting wins as +10 and losses as −1. However, like here, changes to the policy itself, at the level of “do the cooperative thing” versus “don’t do the cooperative thing”, are always ultimately zero-sum – you can’t increase both P(A) and P(¬A).
I agree that the training made the model “defective”.
This is a pretty generic feature of standard RL algorithms in these multi-agent settings (see for instance Bell et al. 2021). (However, note that in more realistic environments we’d expect cooperative dynamics to occur alongside some main task at which the models would get better over time, and so aggregate performance would probably still go up.)
Intuitively, it’s because these algorithms work in a CDT-ish way, making the policy perform better assuming that the other copies of the policy are held fixed; the policy at the end would indeed perform better than the policy at the beginning against the same opponent. (If you were training a model to play chess against itself, and draws become more common in better players, then the model’s average win-rate in self-play would go down!)
If you want to target average performance in self-play, you could instead reinforce according to some team’s aggregate reward, as labs probably do in subagent training. In standard PD set-ups, this would mean that C/C gets reinforced more than C/D, since aggregate reward is higher for the former. (We discuss some dangers associated with subagent training here.)
Here’s a version of the first figure for the 39 that Qwen gets correct among the easiest 123; notably now 10⁄39 reconstructions are better than the rock, but the other 75% still aren’t.
Even from the original chart, though, I think “the problems fly over Qwen’s head” isn’t my default hypothesis; since yellow < orange < red for Qwen in that chart (and this one), it’s clearly getting some signal from the differences between the problems (unlike the second figure, where we vary the key number in the problem statement, where Qwen gets ~no signal from this because yellow = orange = red approximately).
For your second point: if you actually compute optimal [rock I] here, we can tell that it’s badly overfit: working with the 123 easiest, if you take [rock I] which is optimal for the other 122 and apply that to your problem of interest, doing this for every problem gives you a mean cosine FVU of 2.05 (i.e., way worse than not doing this). But leaving one out (indeed, leaving half out) makes little difference when instead we just take corrected_i to be unit(unit(recon_i) – mean(unit(recons)) + mean(unit(originals))), i.e., we just correct for the mean bias:
That’s some improvement! In particular, more than 75% of Qwen’s activations now do better than baseline. (Using the fully-optimal one instead just moves yellow and red each down relative to this, and in particular every yellow does better than baseline, although the best yellows don’t move down much; but again, the fully-optimal one is badly overfit.)
So, we knew that Qwen’s NLA was able to distinguish between different problems (even though it ~ignores the numbers), but overall has worse reconstruction error than an optimal rock. But we’ve learned, from a version of the second experiment you suggest, that a large component of its bad reconstruction error can be explained by a single bias direction for these problems.
The certainty of one million is qualitatively different from a 99% chance of getting at least one million with a 1% chance of getting nothing. That 1% of nothing looms large because of what it means in context: you are giving up a sure million for a gamble that could leave you with nothing.
I really liked your Ellsberg section, and it changed my mind somewhat. But here’s a quibble with the Allais section: “certainty” / “sure” / “could” seem to point to a strict guarantee, not just full probability (“almost sure”); and to the extent that “sure” and “99%” feel “qualitatively different”, it feels like “almost sure” lands closer to “99%” than to “sure”.
More explicitly: suppose that I’m going to flip a fair coin countably many times, and read off a real between 0 and 1 from the resulting sequence; I offer you the choice between (i) 1 million for sure, or (ii) 1 million if the number is an irrational below 0.9, and 5 million if the number is an irrational above 0.9 (so that you get nothing if the result is a rational number). It feels like your defense of Allais preferences would also license picking (i) over (ii): if you can talk yourself out of picking (i) here – “I mean, it could happen in principle, but it’s extremely unlikely” – then it feels like you should be able to talk yourself out of the Allais preferences. But this seems much worse, since you’re giving up the 10% chance of 5 million over the possibility of a probability zero event. So your defense of Allais preferences feels like it overgenerates.Further, we could run a version of Allais where your first option is just 1 million with probability 1, but not for sure (e.g., you get the million only if my coin produces an irrational sequence). But again, it seems somewhat weird to give up your Allais preferences once I make this modification. So your defense of Allais preferences feels like it undergenerates, too.
(To the extent that I’m sympathetic to Allais preferences, I’m tempted to go along with these weird-seeming conclusions; but I’m not very sympathetic, so maybe I’m not the best judge.)
[superseded by Chi’s comment]