I (Evan Hubinger) am a Research Fellow at MIRI working on inner alignment for amplification.
See: “What I’ll doing at MIRI.”
Pronouns: he/him/his
Email: evanjhub@gmail.com
Selected work:
evhub(Evan Hubinger)
Karma: 3,321
Yeah, I agree—the example should probably just be changed to be about an imitative amplification agent or something instead.
I think that trying to encourage myopia via behavioral incentives is likely to be extremely difficult, if not impossible (at least without a better understanding of our training processes’ inductive biases). Krueger et al.’s “Hidden Incentives for Auto-Induced Distributional Shift” is a good resource for some of the problems that you run into when you try to do that. As a result, I think that mechanistic incentives are likely to be necessary—and I personally favor some form of relaxed adversarial training—but that’s going to require us to get a better understanding of what exactly it looks for an agent to be myopic or not so we know what the overseer in a setup like that should be looking for.
Possibly relevant here is my transparency trichotomy between inspection transparency, training transparency, and architectural transparency. My guess is that inspection transparency and training transparency would mostly go in your “active transparency” bucket and architectural transparency would mostly go in your “passive transparency” bucket. I think there is a position here that makes sense to me, which is perhaps what you’re advocating, that architectural transparency isn’t relying on any sort of path-continuity arguments in terms of how your training process is going to search through the space, since you’re just trying to guarantee that the whole space is transparent, which I do think is pretty good desideratum if it’s achievable. Imo, I mostly bite the bullet on path-continuity arguments being necessary, but it definitely would be nice if they weren’t.
Open Problems with Myopia
Fwiw, I also agree with Adele and Eliezer here and just didn’t see Eliezer’s comment when I was giving my comments.
Sure—by that definition of realizability, I agree that’s where the difficulty is. Though I would seriously question the practical applicability of such an assumption.
Perhaps I just totally don’t understand what you mean by realizability, but I fail to see how realizability is relevant here. As I understand it, realizability just says that the true model has some non-zero prior probability—but that doesn’t matter (at least for the MAP, which I think is a better model than the full posterior for how SGD actually works) as long as there’s some deceptive model with greater prior probability that’s indistinguishable on the training distribution, as in my simple toy model from earlier.
Yeah, that’s a fair objection—my response to that is just that I think that preventing a model from being able to distinguish training and deployment is likely to be impossible for anything competitive.
Here’s a simple toy model. Suppose you have two agents that internally compute their actions as follows (perhaps with actual argmax replaced with some smarter search algorithm, but still basically structured as below):
Then, comparing the K-complexity of the two models, we get
and the problem becomes that both and will produce behavior that looks aligned on the training distribution, but has to be much more complex. To see this, note that essentially any will yield good training performance because the model will choose to act deceptively during training, whereas if you want to get good training performance without deception, then has to actually encode the full objective, which is likely to make it quite complicated.
It’s not that simulating is difficult, but that encoding for some complex goal is difficult, whereas encoding for a random, simple goal is easy.
I don’t think I would take that bet—I think the specific question of what UTM to use does feel more likely to be off-base than other insights I associate with UDASSA. For example, some things that I feel UDASSA gets right: a smooth continuum of happeningness that scales with number of clones/amount of simulation compute/etc., and simpler things being more highly weighted.
Yeah—I think I agree with what you’re saying here. I certainly think that UDASSA still leaves a lot of things unanswered and seems confused about a lot of important questions (embeddedness, uncomputable universes, what UTM to use, how to specify an input stream, etc.). But it also feels like it gets a lot of things right in a way that I don’t expect a future, better theory to get rid of—that is, UDASSA feels akin to something like Newtonian gravity here, where I expect it to be wrong, but still right enough that the actual solution doesn’t look too different.
FWIW I have decent odds on “a thicker computer (and, indeed, any number of additional copies of exactly the same em) has no effect”, and that’s more obviously in contradiction with UDASSA.
Absolutely no effect does seem pretty counterintuitive to me, especially given that we know from QM that different levels of happeningness are at least possible.
Like, I continue to have the dualing intuitions “obviously more copies = more happening” and “obviously, setting aside how it’s nice for friends to have backup copies in case of catastrophe, adding an identical em of my bud doesn’t make the world better, nor make their experiences different (never mind stronger)”. And, while UDASSA is a simple idea that picks a horse in that race, it doesn’t… reveal to each intuition why they were confused, and bring them into unison, or something?
I think my answer here would be something like: the reason that UDASSA doesn’t fully resolve the confusion here is that UDASSA doesn’t exactly pick a horse in the race as much as it enumerates the space of possible horses, since it doesn’t specify what UTM you’re supposed to be using. For any (computable) tradeoff between “more copies = more happening” and “more copies = no impact” that you want, you should be able to find a UTM which implements that tradeoff. Thus, neither intuition really leaves satisfied, since UDASSA doesn’t actually take a stance on how much each is right, instead just deferring that problem to figuring out what UTM is “correct.”
For another instance, I weakly suspect that “running an emulation on a computer with 2x-as-thick wires does not make them twice-as-happening” is closer to the truth than the opposite, in apparent contradiction with UDASSA.
I feel like I would be shocked if running a simulation on twice-as-thick wires made it twice as easy to specify you, according to whatever the “correct” UTM is. It seems to me like the effect there shouldn’t be nearly that large.
The only reason that sort of discarding works is because of decoherence (which is a probabilistic, thermodynamic phenomenon), and in fact, as a result, if you want to be super precise, discarding actually doesn’t work, since the impact of those other eigenfunctions never literally goes to zero.
I think I basically agree with all of this, though I definitely think that the problem that you’re pointing to is mostly not about the Born rule (as I think you mostly state already), and instead mostly about anthropics. I do personally feel pretty convinced that at least something like UDASSA will serve the test of time on that front—it seems to me like you mostly agree with that, but just think that there are problems with embededness + figuring out how to properly extract a sensory stream that still need to be resolved, which I definitely agree with, but still expect the end result of resolving those issues to look UDASSA-ish.
I also definitely agree that there are really important hints as to how we’re supposed to do things like anthropics that we can get from looking at physics. I think that if you buy that we’re going to want something UDASSA-ish, then one way in which we can interpret the hint that QM is giving us is as a hint as to what our Universal Turing Machine should be. Obviously, the problem with that is that it’s a bit circular, since you don’t want to choose a UTM just by taking a maximum likelihood estimate, otherwise you just get a UTM with physics as a fundamental operation. I definitely still feel confused about the right way to use physics as evidence for what our UTM should be, but I do feel like it should be some form of evidence—perhaps we’re supposed to have a pre-prior or something here to handle combining our prior beliefs about simple UTMs with our observations about what sorts of UTM properties would make the physics that we find ourselves in look simple.
It’s also worth noting that UDASSA pretty straightforwardly predicts the existence of happeningness dials, which makes me find their existence in the real world not all that surprising.
Also, it’s not really either here not there, but as an aside I feel like this sort of discussion is where the meat of not just anthropics, but also population ethics is supposed to be—if we can figure out what the “correct” anthropic distribution is supposed to be (whatever that means), then that measure should also clearly be the measure that you use to weight person-moments in your utilitarian calculations.
I think that “think up good strategies for achieving [reward of the type I defined earlier]” is likely to be much, much more complex (making it much more difficult to achieve with a local search process) than an arbitrary goal X for most sorts of rewards that we would actually be happy with AIs achieving.
A’s structure can just be “think up good strategies for achieving X, then do those,” with no explicit subroutine that you can find anywhere in A’s weights that you can copy over to B.
A is sufficiently powerful to select M which contains the complex part of B. It seems rather implausible that an algorithm of the same power cannot select B.
A’s weights do not contain the complex part of B—deception is an inference-time phenomenon. It’s very possible for complex instrumental goals to be derived from a simple structure such that a search process is capable of finding that simple structure that yields those complex instrumental goals without being able to find a model with those complex instrumental goals hard-coded as terminal goals.
Yes; episode is correct there—the whole point of that example is that, by breaking the episodic independence assumption, otherwise hidden non-myopia can be revealed. See the discussion of the prisoner’s dilemma unit test in Krueger et al.’s “Hidden Incentives for Auto-Induced Distributional Shift” for more detail on how breaking this sort of episodic independence plays out in practice.