I (Evan Hubinger) am a Research Fellow at MIRI working on inner alignment for amplification.
See: “What I’ll doing at MIRI.”
Pronouns: he/him/his
Email: evanjhub@gmail.com
Selected work:
evhub(Evan Hubinger)
Karma: 3,624
Yeah, sorry, I wasn’t clear here—I meant that, rather than reasoning about the complexity of individual pieces / stages and then adding them all up at the end, I am instead simulating out the equations
Sure, makes sense—theoretically, that should be isomorphic.
I want to note that since is chosen randomly, it isn’t “choosing” the condition on ; rather the wide distribution over leads to a wide distribution over possible conditions on . But I think that’s what you mean.
This seems like a case where I’m using the more constructive formulation of simulating out the equations and you’re thinking about in a more complexity-oriented framing. Of course, again, they should be equivalent.
By our bijection assumption, the parameters in must be identical to the parameters in .
I’m not sure what you mean by this part— and are just different heads, not entirely different models, so I’m not sure what you mean by “the parameters in .” I don’t think that a bijection assumption between weights and single-head outputs really makes sense in this context. I also definitely would say that if and were separate models such that they couldn’t reuse weights between them, then none of the complexity arguments that I make in the post would go through.
These constraints are necessary and sufficient to satisfy the overall constraint that , and therefore any other parameters in are completely unconstrained and are set according to the original neural net prior.
I’m happy to accept that there are ways of setting (e.g. just make and identical) such that the rest of the parameters are unconstrained and just use the neural net prior. However, that’s not the only way of setting —and not the most complexity-efficient, I would argue. In the defender’s argument, sets all the head-specific parameters for both and to enforce that computes and computes , and also sets all the shared parameters for everything other than the human model, while leaving the human model to , thus enforcing that specify a human model that’s correct enough to make without having to pay any extra bits to do so.
Hmm, I’m not thinking about the complexity part at all right now; I’m just thinking mechanically about what is implied by your equations.
The only difference between this setup and normal ML is the prior/complexity—you still have the ability to learn all the same functions, it’s just that some are more/less likely now.
though of course to check the condition you need to look at both parameters, so you can view this as the first set of parameters specifying a condition on the second set of parameters
Yep, that’s exactly right.
Imagine there was a bijection between model parameters and resulting function. (I’m aware this is not at all true.) In that case it seems like you are enforcing the constraint that the two heads have identical parameters.
That’s definitely not what should happen in that case. Note that there is no relation between and or and —both sets of parameters contribute equally to both heads. Thus, can enforce any condition it wants on by leaving some particular hole in how it computes and and forcing to fill in that hole in such a way to make ’s computation of the two heads come out equal.
It seems like at this point your prior is “generate parameters randomly under the constraint that the two heads are identical”
That’s not what the prior looks like—the prior is more like “generate parameters that specify some condition, then sample parameters that make that condition true.” Thus, you don’t need to pay for the complexity of satisfying the condition, only the complexity of specifying it (as long as you’re content with the simplest possible way to satisfy it). This is why the two-step nature of the algorithm is necessary—the prior you’re describing is what would happen if you used a one-step algorithm rather than a two-step algorithm (which I agree would then not do anything).
(A possible objection would be: “real-world foresighted planning” isn’t a separate thing that adds to model complexity, instead it naturally falls out of other capabilities that are necessary for postdiction like “building predictive models” and “searching over strategies” and whatnot. I think I would disagree with that objection, but I don’t have great certainty here.)
Yup, that’s basically my objection.
Sure, that’s fair. But in the post, you argue that this sort of non-in-universe-processing won’t happen because there’s no incentive for it:
It seems like there’s no incentive whatsoever for a postdictive learner to have any concept that the data processing steps in the algorithm have any downstream impacts, besides, y’know, processing data within the algorithm. It seems to me like there’s a kind of leap to start taking downstream impacts to be a relevant consideration, and there’s nothing in gradient descent pushing the algorithm to make that leap, and there doesn’t seem to be anything about the structure of the domain or the reasoning it’s likely to be doing that would lead to making that leap, and it doesn’t seem like the kind of thing that would happen by random noise, I think.
However, if there’s another “why” for why the model is doing non-in-universe-processing that is incentivized—e.g. simplicity—then I think that makes this argument no longer hold.
I mean, I guess it depends on your definition of “unrelated to any anticipated downstream real-world consequences.” Does the reason “it’s the simplest way to solve the problem in the training environment” count as “unrelated” to real-world consequences? My point is that it seems like it should, since it’s just about description length, not real-world consequences—but that it could nevertheless yield arbitrarily bad real-world consequences.
Oh, I guess you’re saying something different: that even a deceptive mesa-optimizer which is entirely doing within-universe processing is nevertheless scary. So that would by definition be an algorithm with the property “no operation in the algorithm is likelier to happen vs not happen specifically because of anticipated downstream chains of causation that pass through things in the real world”.
Yep, that’s right.
So I can say categorically: such an algorithm won’t hurt anyone (except by freak accident), won’t steal processing resources, won’t intervene when I go for the off-switch, etc., right?
No, not at all—just because an algorithm wasn’t selected based on causing something to happen in the real world doesn’t mean it won’t in fact try to make things happen in the real world. In particular, the reason that I expect deception in practice is not primarily because it’ll actually be selected for, but primarily just because it’s simpler, and so it’ll be found despite the fact that there wasn’t any explicit selection pressure in favor of it. See: “Does SGD Produce Deceptive Alignment?”
I mostly agree with what Paul said re using various techniques to improve the evaluation of to ensure you can test it on more open-ended questions. That being said, I’m more optimistic that, if you can get the initial training procedure right, you can rely on generalization to fill in the rest. Specifically, I’m imagining a situation where the training dataset is of the narrower form you talk about such that and always agree (as in Step 3 here)—but where the deployment setting wouldn’t necessarily have to be of this form, since once you’re confident that you’ve actually learned and not e.g. , you can use it for all sorts of things that wouldn’t ever be in that training dataset (the hard part, of course, is ever actually being confident that you did in fact learn the intended model).
(Also, thanks for catching the typo—it should be fixed now.)
By and large, we expect learning algorithms to do (1) things that they’re being optimized to do, and (2) things that are instrumentally useful for what they’re being optimized to do, or tend to be side-effect of what they’re being optimized to do, or otherwise “come along for the ride”. Let’s call those things “incentivized”. Of course, it’s dicey in practice to declare that a learning algorithm categorically won’t ever do something, just because it’s not incentivized. Like, we may be wrong about what’s instrumentally useful, or we may miss part of the space of possible strategies, or maybe it’s a sufficiently simple thing that it can happen by random chance, etc.
In the presence of deceptive alignment, approximately any goal is possible in this setting, not just the nearby instrumental proxies that you might be okay with. Furthermore, deception need not be 4th-wall-breaking, since the effect of deception on helping you do better in the training process entirely factors through the intended output channel. Thus, I would say that within-universe mesa-optimization can be arbitrarily scary if you have no way of ruling out deception.
Answering questions honestly instead of predicting human answers: lots of problems and some solutions
(Moderation note: added to the Alignment Forum from LessWrong.)
(Moderation note: added to the Alignment Forum from LessWrong.)
There are other models than the discontinuous/fast takeoff model under which alignment of the first advanced AI is critical, e.g. a continuous/slow but homogenous takeoff.
(Moderation note: added to the Alignment Forum from LessWrong.)
(Moderation note: added to the Alignment Forum from LessWrong.)
(Moderation note: added to the Alignment Forum from LessWrong.)
(Moderation note: added to Alignment Forum from LessWrong.)
Though they’re both somewhat outdated at this point, there are certainly still some interesting concrete experiment ideas to be found in my “Towards an empirical investigation of inner alignment” and “Concrete experiments in inner alignment.”
Have you seen “Growing Neural Cellular Automata?” It seems like the authors there are trying to do something pretty similar to what you have in mind here.
Yep, that’s what I mean.
Note that conditioning on the part-which-shares-weights is definitely not what the prior is doing—the only conditioning in the prior is θ2 conditioning on θ1. If we look at the intended model, however, θ1 includes all of the parts-which-don’t-share-weights, while θ2 is entirely in the part-which-shares-weights.
Technically, I suppose, you can just take the prior and condition on anything you want—but it’s going to look really weird to condition on the part-which-shares-weights having some particular value without even knowing which parts came from θ1 and which came from θ2.
I do agree that, if θ1 were to specify the entire part-which-shares-weights and leave θ2 to fill in the parts-which-don’t-share-weights, then you would get exactly what you’re describing where θ2 would have a doubly-strong neural net prior on implementing the same function for both heads. But that’s only one particular arrangement of θ1—there are lots of other θ1s which induce very different distributions on θ2.
Note that the inputs to f+,f− are deduced statements, not raw data. They are certainly different functions over the space of all possible deduced statements—but once you put a correct world model in them, they should produce equivalent X×Q→A maps.