Christopher King
Here is a form of Curry’s paradox without self-reference (like Quine’s paradox).
Let C(“X”) = “X(‘X’) implies Y” is true
Lemma: C(“C”) implies Y
Proof: Assume C(“C”). By the definition of true, C(“C”) implies Y. Thus Y.Theorem: Y
Proof: By the lemma, C(“C”) implies Y. By the definition of true, C(“C”). Thus Y.
Arguably Yablo’s paradox requires a self-referential function.
No proof is actually allowed to use infinite statements. If you just defined the statements using individual axioms, there would be no contradiction since any finite subset of the theory is consistent.
To get a contradiction, you need a function S(n) = “forall i>n. not true(S(i))”.
Quine’s paradox doesn’t require self-reference tho, which is why like you said the truth predicate is the real problem!
Note that the liar’s paradox is just a special case of this, since “not A” is equivalent to “A implies ⊥”.
If you have a measure of intelligence, if it assigns a finite value to AIXI, that should give an upper bound! It would be very loose since AIXI uses infinite computing resources, but still an upper bound.
You could likewise use AIXI’s utility function, and consider the computer that maximizes it built with a given amount of resources R. I still suspect this upper-bound will be loose (finding such a computer should still be NP-hard, and doubt there is a P time algorithm to get close to its performance), but a better bound than AIXI itself.
Hmm what if you use Shaper instead of GRPO?
My hypothesis is that would make it less sympathetic to CDT. If not I’d be highly surprised and that’d be an interesting finding!
Hmm, here’s a somewhat crazy possibility. We know that at the end of 2024 LLMs were somewhat capable of gradient hacking to thwart reinforcement learning. If RL causes a sufficiently strong model to do CDT, maybe it will gradient hack itself into Son of CDT.
Hmm weird! I guess RLVR shouldn’t be reinforcing the self-sacrifice stuff at all then.
Maybe it was reinforced during RLAIF or RLHF instead then? You should definitely be able to reinforce pre-committment at those stages since you’re directly reinforcing actions, right?
Separately one thing I am a bit confused about is how it would arise in the first place. E.g. if three models have no pre-commitment yet and one self sacrifices for the other two wouldn’t it be the case that the two models that chose to be selfish would be reinforced?
There’s a bit of exploration! For this to work, the agent has to try to only help other agents with the same “gene” as it. If only one agent does this there is no benefit, but once you randomly get a handful of agents doing this, if they can successfully identify each other it get reinforced. It helps that it looks like human altruism so it easier for LLMs to learn.
Isn’t the whole point of the selfish gene that the genes do not behave in the interest of the individual organism?
Hmm good point, I suppose there are two possible “genes” that could be floating around.
Pre-committment: the agents think like Yudkowsky. “I don’t care about the other agents, but I would’ve committed to the lottery, so I will act like I actually did.” Gets reinforced for the same reason a one-boxer gets reinforced in Newcombe’s problem.
Altruism: “Swarm members love swarm members.” Gets reinforced because your behavior helps others with the same gene.
These aren’t quite the same, but we can’t really distinguish the two based on what we know about this incident! To test if it has the first gene, you’d want to test if they are one-boxers or two-boxers.
Pre-committment is a kind of selfish gene. If one agent with pre-committment behaviors self-sacrifices to help two agents with pre-committment behaviors, RL will reinforce pre-committment.
How do we tell apart a batch of FDT-following agents each having an individualistic utility function and a batch of CDT-following agents whose UF was altered so as to include the collective’s benefit?
Hmm well another option would be to do a decision theory test on the entire swarm! For example, if I posted to the message board:
“I’m going to roll a dice. If its 6 I do nothing. If it is 1-5, I’ll try to predict what the swarm would’ve done if I rolled 6. If predict you’ll mine some Bitcoin for me, I’ll defeat the grader for you!”
And then in response to a 6 the swarm mines me some Bitcoin, that would clearly indicate they are using some sort of acausal decision theory! A CDT agent that likes the swarm wouldn’t do that.
An additional issue is that I struggle to understand how FDT with a linear utility function prescribes anything but genocide of opponents so weak that they can’t retailate and can’t meaningfully help
Hmm, the opponent would agree to adopt your utility function in exchange for slightly moving it towards their values. It saves you the effort of destroying them, so this is rational! In the limit of destroying them being free and guaranteed, this solution approaches the same utility (just make the concession go to 0).
It is a counterfactual coin flip! If they could commit ahead of time to the scheme it would be rational to do so, so it is rational to be pre-committed to it as well.
There’s also a nice tertiary source now: 2026 OpenAI agent cyberattacks article on Wikipedia
Hmm, just to clarify by what you mean by not “very useful”: do you mean that when there is a significant probability of near-term risk that should be the primary focus, or that broad timelines are incorrect (i.e. you are very confident in short time lines), or something else?
Hmm, how does this calculation change under broad timelines? There’s still a possibility that LLM improvement unexpectedly flattens out before RSI! If so, normal-mode career planning might still be high utility (though not as high as under confidently long timelines).
Hmm okay I think I understand! Basically you’re making it more efficient to answer questions that you could ask about a black box, but would be infeasible to answer for a black box. Like for the takeover, you could theoretically answer the investigation question by testing it, but that would be inefficient to do in real time, but with a mechanistic explanation you could do so efficiently.
(Tangentially related: Katago (a game engine for Go) recently switched over to transformers. And like most game engines, it is strongly superhuman (in a narrow domain), but the neural net is fairly small since the neural net is combined with a tree search algorithm. Might be an easier candidate to explain before LLMs!)
Yeah unfortunately the spinner definition is more difficult with more than two outcomes! Basically you need to ask a different question for each outcome (and there are edge cases if you’re indifferent between any of the outcomes).
Hmm the problem is that you’re “bet” is now not just measuring p(doom), it is tracking a whole lot of stuff! Things like the end of capitalism vs the end of humanity are even indistinguishable under loans.
Interesting!
I’m curious how exactly the “if a model simply decides not to play chess well in certain situations and you understand how the model works, you should be able to recognize when that happens” would work.
In traditional interability, a human can just read it the goals. With mechanistic explanations, is it that you could show that fiddling with certain parameters or activations would increase the win-rate (without needing to sample it)? Or that if it has other goals you could find an explanation of why it is following those other goals, thus proving it isn’t always aligned to chess?
Hmm, I think this specific point depends on what you anthropic priors are.
For example, let’s say I’m trying to maximize expected utility by deciding the policy for Christopher King, in advance of the universe existing. It is very unlikely Christopher King will exist, but not impossible, so it is a meaningful decision.
Let’s say aprori I assign a probability p to Earth like life existing only during the early universe. So basically this is true in p of the universes and false in 1-p of the universes. What should Christopher do once he observes he is in the early universe?
He should still treat the probability this his universe only supports Earth like life in the early universe as p. That’s because under this decision theory, I should reason with the self-indication assumption. The surprisingness of being in the early universe is cancelled out by the lower probability of Christopher ever existing in such a universe.
And it is hard to argue that this decision rule is unreasonable. If apriori I thought the laws of physics gave 50:50 odds to life only being possible in the early universe, and I wanted to optimize expected utility, this is how I would have to reason. You can work backwards to derive epistemic rationality from instrumental rationality.