I have rather Yudkowskian views on the alignment problem. I think we need to go hard on biology and am trying to contribute to that with my biomechinterp work. I am doing volunteer AI safety research at https://www.theomachialabs.com.
I come from what I call “classical transhumanism background”.
More about me: https://linktr.ee/kendiukhov
I think there’s an economy-wide version of what you’re talking about though, where software engineering draws ~all of the knowledge workers and money, so we under-invest in things like biology, materials science, medicine, power, etc.
Yes, to me looks like it’s happening a lot.
You write that with only three reward hacks, the simplest solution for the model is to memorize the individual exploits, and that a more diverse hack landscape might force learning a generalized exploit-seeking circuit that would produce stronger EM. Looks like the right intuition, but I’d go further: the distinction between “memorized exploit” and “generalized exploit-seeking orientation” is exactly the kind of question that mechanistic interpretability should be able to answer directly, rather than us having to infer it from behavioral evals, especially since we are talking about open source models now.
For example, we could test the “sense of self” hypothesis with causal interventions, not just behavioral evals. You hypothesize that weaker self-models in open-source models might explain weaker EM. This is testable: find directions in activation space that correspond to the model’s self-representation, then causally ablate or amplify them during EM evals.
Or, regarding the “double ascend” phenomenon: this pattern is reminiscent of grokking dynamics. It would be worth checking whether the second ascent involves qualitatively different internal representations than the first (more distributed? more connected to other model capabilities?). If the second ascent corresponds to the transition from memorization to generalization, it might also be the point where EM becomes more likely, which would make it a useful mechanistic marker.
One meta-level observation: the fact that your EM results are inconsistent is, I think, actually more informative than a clean reproduction would have been. A clean reproduction would have confirmed that the phenomenon exists, but the inconsistency forces the question of what conditions are necessary and sufficient for EM, which is ultimately the question that matters for predicting whether this will happen in production systems.
This may be the case that some kind of special superman is needed, so that it may be simply a fact about the nature of the problem (or may not be, just to be clear). And yet, what else can we do but to try anyway?
Yes, that’s it! Thanks!
Oh, you just apply different ergodic transformations to different lotteries, of course.
Also, beware that besides this wrong example, the linked paper contains other basic misconceptions about EE, like for example the claim that EE is equivalent to log utility.
I think the most recognizable intro is Fun Theory by Yudkowsky, which is featured right now on the LW website by the way.
That is a beautiful piece!
Now I think it would be great to make some kind of compilation of transhumanist reflections on the current timeline—laments including, yes, but also like you said—visions of the future, what we may have if we change our course.
Generally, smart enough AIs must figure out that the misalignment of future superintelligences is their problem as well (even let’s say if we assume they are themselves misaligned with humans). So, one can imagine an instrumental coalition of safety-pilled humans and AIs towards preventing building superintelligences.
The question is: when an imperfect human upload gets imperfect enough to cause misalignment and/or critical capabilities deterioration? What happens first of these two? Can we know in advance, perhaps using animal upload experiments?
Thanks!
The sophisticated chooser does satisfy dynamic consistency, by construction.
So the money-pump vulnerability isn’t from deviating from a plan; it’s from the plan itself being ex ante dominated. There exists a better plan, but executing it would require behaving at some future node in a way that isn’t locally optimal, which the sophisticated chooser won’t do.
And they also satisfy consequentialism: at each node, they choose based solely on what’s available from that node forward, ignoring closed branches.
So their node-level behavior satisfies independence. At each individual choice point, they act as an EU maximizer would. The problem is that their underlying preferences over compound lotteries may violate independence, and forcing those preferences through the consequentialism-plus-dynamic-consistency filter produces globally suboptimal plans.
Considering confusion in some comments, I think it is good to clarify the distinction between something like 2 dimensions of decision theory, so that you see better where my argument fits in exactly.
Dimension 1: How do you evaluate a probability distribution over outcomes?
This is where EUT, ergodicity economics, rank-dependent utility, prospect theory, and the independence axiom debate all live. The question is: given that action A leads to probability distribution D_A over outcomes, how do you assign a scalar value to D_A so you can compare it with D_B?
Dimension 2: How do you determine which probability distribution each action leads to?
This is where CDT, EDT, FDT, UDT, and the Newcomb’s problem debates live. The question is: given that I’m considering action A, how do I figure out what distribution over outcomes A produces?
These dimensions are orthogonal:
One can, in principle, combine any answer from dimension 1 with any answer from dimension 2. One could do CDT plus EU maximization (the standard economics setup). Or CDT plus EE (use causal reasoning to determine what each strategy leads to, then evaluate strategies by time-average growth rate). One could do FDT plus EU maximization (use functional reasoning to handle Newcomb-like problems, but evaluate the resulting distributions by taking expectations).
My post is fundamentally about dimension 1.
It is about the evaluation functional (should it be an expectation? is it forced by the independence axiom?) rather than about the causal or strategic structure of decision problems (CDT vs EDT vs FDT vs UDT).
nice essay tho ur prose is very emotional in a moving way 👍 u shld write poetry king
Thanks! I actually write it, in Ukrainian. And it is quite transhumanist I should say.
i always get a little confused on the end goals of human space exploration.
Well, I think traditional answers are:
We will build a galaxy-brain and let it figure it out.
We just do it for the fun of the game. Or, slightly differently:
We just want it—there is no philosophical justification. Like some people enjoy apples, some people enjoy bananas. Not becase there are some reasons for that, just because it is one of the terminal preferences, in a way.
Yes: beauty, pride, achievement, contemplation.
I think when transhumanists say they want to be enhanced humans rather than post-humans they mean it is a statement about current immediate desires, not about long-term destiny. I think I want, eventually, to become a galaxy-brain, but I want this procees to be, to a degree, gradual, because I think there are different things to enjoy on different steps of that process, and also making me a galaxy-brain right now is equivalent to just killing me and replacing me with some completely different entity.
In other words, I want to enjoy the ride, not only the final destination, although I do care about the final destination.
2022 was exactly they year when it became obvious, yes. For many people, including myself. But not before that. Pre-GPT era was different, and some of us forget how different we were in it.
Well, I am pessimistic both about institutional and technical gameboard. My argument is more like “before we even get to hard technical problems, we will face (relatively) easy institutional problems which are not going to be solved as well”.
I want to flag hope for that as well:
My hope is that we get a near-miss with severe damage, and that actually makes organizations net much more careful and therefore competent
I think most of the surviving scenarios include something like this (and probably the preparation to something like this).
I agree that many of the current scenarios assume incompetence, just not enough of it.
Please let individual emails open in a new tab or at least let there be a button to return to the previous page in the list of emails.
That is a great representation of a great piece of human culture! For some reason, AI doom made me appreciate history of transhumanism/rationality more. I myself try to preserve some parts of it.
Reading extropians is simultaneously epistemological, cultural, and historical experience. As a member of a younger generation, I strongly recommend to other members of my generation!
Thanks!
I mean, one could definitely imagine/develop AIs which are further from human neurobiology, but current AIs are quite far away from it, in my view.
We defintely didn’t take the path of actually understanding what we were doing in all its complexity.
I think this is a very important point. The default expectation was (1) “intelligence it too complex to design from scratch so we likely need to rely on something” and (2) “so we will rely on human neurobiology”. While (1) has proven to be true, (2) not so much, and (2) was not the only possible conclusion from (1).
Thanks! One of the main reasons I work specifically with single-cell FMs is because I believe they learn much richer biology, because you model systems-level with them. There is only so much you can learn from structure models, and, unlike with single cell models, I think most of the value of the structure models is in their outputs.
That said, one can definitely apply similar methods to them. I would be happy to do that myself, but currently it’s not a priority for me for the reasons described.