I (Evan Hubinger) am a Research Fellow at MIRI. Broadly, I work on inner alignment for prosaic machine learning.
See: “What I’ll doing at MIRI.”
Pronouns: he/him/his
Email: evanjhub@gmail.com
Selected work:
evhub(Evan Hubinger)
Karma: 4,642
Learning the smooth prior
Sure—I just edited it to be maybe a bit less jarring for those who know Greek.
I think this concern is only relevant if your strategy is to do RL on human evaluations of alignment research. If instead you just imitate the distribution of current alignment research, I don’t think you get this problem, at least anymore than we have it now—and I think you can still substantially accelerate alignment research with just imitation. Of course, you still have inner alignment issues, but from an outer alignment perspective I think imitation of human alignment research is a pretty good thing to try.
Actions are just language outputs—and since we ask for an action that describes a pseudo-input, hopefully we should be able to interpret it that way.
The reason other possible reporter heads work is because they access the model’s concept for X and then do something with that (where the ‘doing something’ might be done in the core model or in the head).
I definitely think there are bad reporter heads that don’t ever have to access X. E.g. the human imitator only accesses X if X is required to model humans, which is certainly not the case for all X.
I don’t think
E.g. simplicity of explanation of a particular computation is a bit more like a speed prior.
I don’t think that the size of an explanation/proof of correctness for a program should be very related to how long that program runs—e.g. it’s not harder to prove something about a program with larger loop bounds, since you don’t have to unroll the loop, you just have to demonstrate a loop invariant.
(Moderation note: added to the Alignment Forum from LessWrong.)
Towards a better circuit prior: Improving on ELK state-of-the-art
SGD is meant as a shorthand that includes other similar optimizers like Adam.
Use the length of the shortest interpretability explanation of behaviours of the model as a training loss for ELK—the idea is that models with shorter explanations are less likely to include human simulations / you can tell if they do.
Maybe this is not the right place to ask this, but how does this not just give you a simplicity prior?
This is not true in the circuit-depth complexity model. Remember that an arbitrary lookup table is O(log n) circuit depth. If my function I’m trying to memorize is f(x) = (x & 1), the fastest circuit is O(1), whereas a lookup table is O(log n).
Certainly, I’m assuming that the intended function is not in O(log n), though I think that’s a very mild assumption for any realistic task.
I think the prior you’re suggesting is basically a circuit size prior. How do you think it differs from that?
Musings on the Speed Prior
One point of data re the expected impact of sanctions: the Moscow stock exchange is down 50% (!), which presumably means investors expect very serious consequences for the Russian economy, likely due to sanctions.
Why? What changed in your understanding of ELK?
I think it’s very strange that this is the work that gets this sort of pushback—of all the capabilities research out there, I think this is some of the best from an alignment perspective. See e.g. STEM AI and Thoughts on Human Models for context on why this sort of work is good.
Thanks! I updated the picture in the post with this one.
One of my hopes with the SERI MATS program is that it can help fill this gap by providing a good pipeline for people interested in doing theoretical AI safety research (be that me-style, MIRI-style, Paul-style, etc.). We’re not accepting public applications right now, but the hope is definitely to scale up to the point where we can run many of these every year and accept public applications.
(Moderation note: added to the Alignment Forum from LessWrong.)
This is correct, but at least in the quote above, the most important distinction is that most RL algorithms propagate credit assignment back across steps but not across episodes.