I (Evan Hubinger) am a Research Fellow at MIRI working on inner alignment for amplification.
See: “What I’ll doing at MIRI.”
Pronouns: he/him/his
Email: evanjhub@gmail.com
Selected work:
evhub(Evan Hubinger)
Karma: 3,845
Interesting! I’d definitely be excited to know if you figure out what it’s doing.
70 steps is not very many—does training converge if you train for longer (e.g. 700, 7000, 70000)?
Also, in addition to regularization making this strategy not very effective, I’d also suspect that hyperparameter tuning would break it as well—e.g. I’d be interested in what happens if you do black-box hyperparameter tuning on the base training process’s hyperparameters after applying meta-learning (though then, to be fair to the meta-learning process, you’d also probably want to do the meta-learning in a setting with variable hyperparameters).
(Moderation node: added to the Alignment Forum from LessWrong.)
I thought this was a great post, and am excited about reading more, but having just skimmed through the rest of the posts in the sequence, I was disappointed to find no discussion of UDASSA, which I personally find to be by far the most compelling theory of anthropics. Is there a reason for that? What do you think about UDASSA as an alternative to either SSA or SIA?
Since it was brought up to me, I also want to clarify that EA Funds can fund essentially anyone, including:
people who have a separate job but want to spend extra time doing an EA project,
people who don’t have a Bachelor’s degree or any other sort of academic credentials,
kids who are in high school but are excited about EA and want to do something,
fledgling organizations,
etc.
Thanks! I hope the post is helpful to you or anyone else trying to think about the type signatures of goals. It’s definitely a topic I’m pretty interested in.
Have you seen Mark and my “Agents Over Cartesian World Models”? Though it doesn’t have any Selection Theorems in it, and it just focuses on the type signatures of goals, it does go into a lot of detail about possible type signatures for agent’s goals and what the implications of those type signatures would be, starting from the idea that a goal can be defined on any part of a Cartesian boundary.
Suppose the intended model is to predict H’s estimate at convergence, and the actual model is predicting H’s estimate at round N for some fixed N larger than any convergence time in the training set. Would you call this an “inner alignment failure”, an “outer alignment failure”, or something else (not an alignment failure)?
I would call that an inner alignment failure, since the model isn’t optimizing for the actual loss function, but I agree that the distinction is murky. (I’m currently working on a new framework that I really wish I could reference here but isn’t quite ready to be public yet.)
It seems that convergence on such questions might take orders of magnitude more time than what M was trained on. What do you think would actually happen if the humans asked their AI advisor to help with a decision like this? (What are some outcomes you think are plausible?)
That’s a hard question to answer, and it really depends on how optimistic you are about generalization. If you just used current methods but scaled up, my guess is you would get deception and it would try to trick you. If we condition on it not being deceptive, I’d guess it was pursuing some weird proxies rather than actually trying to report the human equilibrium after any number of steps. If we condition on it actually trying to report the human equilibrium after some number of steps, though, my guess is that the simplest way to do that isn’t to have some finite cutoff, so I’d guess it’d do something like an expectation over exponentially distributed steps or something.
What’s your general thinking about this kind of AI risk (i.e., where an astronomical amount of potential value is lost because human-AI systems fail to make the right decisions in high-stakes situations that are eventuated by the advent of transformative AI)? Is this something you worry about as an alignment researcher, or do you (for example) think it’s orthogonal to alignment and should be studied in another branch of AI safety / AI risk?
Definitely seems worth thinking about and taking seriously. Some thoughts:
Ideally, I’d like to just avoid making any decisions that lead to lock-in while we’re still figuring things out (e.g. wait to build anything like a sovereign for a long time). Of course, that might not be possible/realistic/etc.
Hopefully, this problem will just be solved as AI systems become more capable—e.g. if you have a way of turning any unaligned benchmark system into a new system that honestly/helpfully reports everything that the unaligned benchmark knows, then as the unaligned benchmark gets better, you should get better at making decisions with the honest/helpful system.
You can talk to EA Funds before applying
(Moderation note: added to the Alignment Forum from LessWrong.)
What would happen if we instead use convergence as the stopping condition (and throw out any questions that take more than some fixed or random threshold to converge)? Can we hope that M would be able to extrapolate what we want it to do, and predict H’s reflective equilibrium even for questions that take longer to converge than what it was trained on?
This is definitely the stopping condition that I’m imagining. What the model would actually do, though, if you, at deployment time, give it a question that takes the human longer to converge on than any question it ever saw in training isn’t a question I can really answer, since it’s a question that’s dependent on a bunch of empirical facts about neural networks that we don’t really know.
The closest we can probably get to answering these sorts of generalization questions now is just to liken the neural net prior to a simplicity prior, ask what the simplest model is that would fit the given training data, and then see if we can reason about what the simplest model’s generalization behavior would be (e.g. the same sort of reasoning as in this post). Unfortunately, I think that sort of analysis generally suggests that most of these sorts of training setups would end up giving you a deceptive model, or at least not the intended model.
That being said, in practice, even if in theory you think you get the wrong thing, you might still be able to avoid that outcome if you do something like relaxed adversarial training to steer the training process in the desired direction via an overseer checking the model using transparency tools while you’re training it.
Regardless, the point of this post, and AI safety via market making in general, though, isn’t that I think I have a solution to these sorts of inner-alignment-style tricky generalization problems—rather, it’s that I think AI safety via market making is a good/interesting outer-alignment-style target to push for, and that I think AI safety via market making also has some nice properties (e.g. compatibility with per-step myopia) that potentially make it easier to do inner alignment for (but still quite difficult, as with all other proposals that I know of).
Now, if we just want to evaluate AI safety via market making’s outer alignment, we can just suppose that somehow we do get a model that just produces the answer that H would at convergence, and ask whether that answer is good. And even then I’m not sure—I think that there’s still the potential for debate-style bad equilibria where some bad/incorrect arguments are just more convincing to the human than any good/correct argument, even after the human is exposed to all possible counterarguments. I do think that the market-making equilibrium is probably better than the debate equilibrium, since it isn’t limited to just two sides, but I don’t believe that very strongly.
Mostly, for me, the point of AI safety via market making is just that it’s another way to get a similar sort of result as AI safety via debate, but that it allows you to do it via a mechanism that’s more compatible with myopia.
Or is this just exploring a potential approximation?
Yeah, that’s exactly right—I’m interested in how an agent can do something like manage resource allocation to do the best HCH imitation in a resource-bounded setting.
Are we including “long speech about why human should give high approval to me because I’m suffering” as an action? I guess there’s a trade-off here, where limiting to word-level output demands too much lookahead coherence of the human, while long sentences run the risk of incentivizing reward tampering. Is that the reason you had in mind?
Yep, that’s the idea.
This argument doesn’t seem to work, because the zero utility function makes everything optimal.
Yeah, that’s fair—if you add the assumption that every trajectory has a unique utility (that is, your preferences are a total ordering), though, then I think the argument still goes through. I don’t know how realistic an assumption like that is, but it seems plausible for any complex utility function over a relatively impoverished domain (e.g. a complex utility function over observations or actions would probably have this property, but a simple utility function over world states would probably not).
(Moderation note: added to the Alignment Forum from LessWrong.)
Sure, but presumably they’ll also say what particular attacks are so hard that current ML models aren’t capable of solving them—and I think that’s a valuable piece of information to have.
Yeah, that’s a great question—I should have talked more about this. I think there are three ways to handle this sort of problem—and ideally we should do some combination of all three:
Putting the onus on the attacker. Probably the simplest way to handle this problem is just to have the attacker produce larger specification breaks than anything that exists in the model independently of the attacker. If it’s the case that whenever the attacker does something subtle like you’re describing the auditor just finds some other random problem, then the attacker should interpret that as a sign that they can get away with less subtle attacks and try larger modifications instead.
Putting the onus on the judge. Another way to address this sort of problem is just to have a better, more rigorous specification. For example, the specification could include a baseline model that the auditor is supposed to compare to but which they can only sample from not view the internals of (that way modifying an individual neuron in the baseline model isn’t trivial for the auditor to detect).
Putting the onus on the auditor. The last way to handle this sort of problem is just to force the auditor to do better. There is a right answer to what behavior exists in the model that’s not natural for language models to usually learn when trained on general webtext, and a valid way to handle this problem is to just keep iterating on the auditor until they figure out how to find that right answer. For example: if all your attacks are fine-tuning attacks on some dataset, there is a correct answer about what sort of dataset was fine-tuned on, and you can just force the auditor to try to find that correct answer.
Is it inconsistent with the original meaning of the term? I thought that the original meaning of inside view was just any methodology that wasn’t reference-class forecasting—and I don’t use the term “outside view” at all.
Also, I’m not saying that “inside view” means “real reason,” but that my real reason in this case is my inside view.
Automating Auditing: An ambitious concrete technical research proposal
That’s a really interesting thought—I definitely think you’re pointing at a real concern with LCDT now. Some thoughts:
Note that this problem is only with actually running agents internally, not with simply having the objective of imitating/simulating an agent—it’s just that LCDT will try to simulate that agent exclusively via non-agentic means.
That might actually be a good thing, though! If it’s possible to simulate an agent via non-agentic means, that certainly seems a lot safer than internally instantiating agents—though it might just be impossible to efficiently simulate an agent without instantiating any agents internally, in which case it would be a problem.
In some sense, the core problem here is just that the LCDT agent needs to understand how to decompose its own decision nodes into individual computations so it can efficiently compute things internally and then know when and when not to label its internal computations as agents. How to decompose nodes into subnodes to properly work with multiple layers is a problem with all CDT-based decision theories, though—and it’s hopefully the sort of problem that finite factored sets will help with.
But by assumption, it doesn’t think it can influence anything downstream of those (or the probability that they exist, I assume).
This is not true—LCDT is happy to influence nodes downstream of agent nodes, it just doesn’t believe it can influence them through those agent nodes. So LCDT (at decision time) doesn’t believe it can change what HCH does, but it’s happy to change what it does to make it agree with what it thinks HCH will do, even though that utility node is downstream of the HCH agent nodes.
I don’t think I agree with this—in particular, it seems like even given the simulation hypothesis, there could still be quite a lot of value to be had from influencing how that simulation goes. For example, if you think you’re in an acausal trade simulation, succeeding in building aligned AI would have the effect of causing the simulation runner to trade with an aligned AI rather than a misaligned one, which could certainly have an “extraordinary impact.”