I have rather Yudkowskian views on the alignment problem. I think we need to go hard on biology and am trying to contribute to that with my biomechinterp work. I am doing volunteer AI safety research at https://www.theomachialabs.com.
I come from what I call “classical transhumanism background”.
More about me: https://linktr.ee/kendiukhov
I think there’s an economy-wide version of what you’re talking about though, where software engineering draws ~all of the knowledge workers and money, so we under-invest in things like biology, materials science, medicine, power, etc.
Yes, to me looks like it’s happening a lot.
You write that with only three reward hacks, the simplest solution for the model is to memorize the individual exploits, and that a more diverse hack landscape might force learning a generalized exploit-seeking circuit that would produce stronger EM. Looks like the right intuition, but I’d go further: the distinction between “memorized exploit” and “generalized exploit-seeking orientation” is exactly the kind of question that mechanistic interpretability should be able to answer directly, rather than us having to infer it from behavioral evals, especially since we are talking about open source models now.
For example, we could test the “sense of self” hypothesis with causal interventions, not just behavioral evals. You hypothesize that weaker self-models in open-source models might explain weaker EM. This is testable: find directions in activation space that correspond to the model’s self-representation, then causally ablate or amplify them during EM evals.
Or, regarding the “double ascend” phenomenon: this pattern is reminiscent of grokking dynamics. It would be worth checking whether the second ascent involves qualitatively different internal representations than the first (more distributed? more connected to other model capabilities?). If the second ascent corresponds to the transition from memorization to generalization, it might also be the point where EM becomes more likely, which would make it a useful mechanistic marker.
One meta-level observation: the fact that your EM results are inconsistent is, I think, actually more informative than a clean reproduction would have been. A clean reproduction would have confirmed that the phenomenon exists, but the inconsistency forces the question of what conditions are necessary and sufficient for EM, which is ultimately the question that matters for predicting whether this will happen in production systems.
This may be the case that some kind of special superman is needed, so that it may be simply a fact about the nature of the problem (or may not be, just to be clear). And yet, what else can we do but to try anyway?
Yes, that’s it! Thanks!
Oh, you just apply different ergodic transformations to different lotteries, of course.
Also, beware that besides this wrong example, the linked paper contains other basic misconceptions about EE, like for example the claim that EE is equivalent to log utility.
I think the most recognizable intro is Fun Theory by Yudkowsky, which is featured right now on the LW website by the way.
That is a beautiful piece!
Now I think it would be great to make some kind of compilation of transhumanist reflections on the current timeline—laments including, yes, but also like you said—visions of the future, what we may have if we change our course.
Generally, smart enough AIs must figure out that the misalignment of future superintelligences is their problem as well (even let’s say if we assume they are themselves misaligned with humans). So, one can imagine an instrumental coalition of safety-pilled humans and AIs towards preventing building superintelligences.
The question is: when an imperfect human upload gets imperfect enough to cause misalignment and/or critical capabilities deterioration? What happens first of these two? Can we know in advance, perhaps using animal upload experiments?
Thanks!
The sophisticated chooser does satisfy dynamic consistency, by construction.
So the money-pump vulnerability isn’t from deviating from a plan; it’s from the plan itself being ex ante dominated. There exists a better plan, but executing it would require behaving at some future node in a way that isn’t locally optimal, which the sophisticated chooser won’t do.
And they also satisfy consequentialism: at each node, they choose based solely on what’s available from that node forward, ignoring closed branches.
So their node-level behavior satisfies independence. At each individual choice point, they act as an EU maximizer would. The problem is that their underlying preferences over compound lotteries may violate independence, and forcing those preferences through the consequentialism-plus-dynamic-consistency filter produces globally suboptimal plans.
Considering confusion in some comments, I think it is good to clarify the distinction between something like 2 dimensions of decision theory, so that you see better where my argument fits in exactly.
Dimension 1: How do you evaluate a probability distribution over outcomes?
This is where EUT, ergodicity economics, rank-dependent utility, prospect theory, and the independence axiom debate all live. The question is: given that action A leads to probability distribution D_A over outcomes, how do you assign a scalar value to D_A so you can compare it with D_B?
Dimension 2: How do you determine which probability distribution each action leads to?
This is where CDT, EDT, FDT, UDT, and the Newcomb’s problem debates live. The question is: given that I’m considering action A, how do I figure out what distribution over outcomes A produces?
These dimensions are orthogonal:
One can, in principle, combine any answer from dimension 1 with any answer from dimension 2. One could do CDT plus EU maximization (the standard economics setup). Or CDT plus EE (use causal reasoning to determine what each strategy leads to, then evaluate strategies by time-average growth rate). One could do FDT plus EU maximization (use functional reasoning to handle Newcomb-like problems, but evaluate the resulting distributions by taking expectations).
My post is fundamentally about dimension 1.
It is about the evaluation functional (should it be an expectation? is it forced by the independence axiom?) rather than about the causal or strategic structure of decision problems (CDT vs EDT vs FDT vs UDT).
Thanks! One of the main reasons I work specifically with single-cell FMs is because I believe they learn much richer biology, because you model systems-level with them. There is only so much you can learn from structure models, and, unlike with single cell models, I think most of the value of the structure models is in their outputs.
That said, one can definitely apply similar methods to them. I would be happy to do that myself, but currently it’s not a priority for me for the reasons described.