Thanks, Richard!
I do think both of those cases fit into the framework fine (unless I’m misunderstanding what you have in mind):
In the first case, we’re training a model in an environment. As it gets more capable, it reaches a point where it can find new, harmful behaviors in some set of situations. Our worries are now that (1) we can’t recognize that behavior as harmful, or (2) we don’t visit those situations during training, but they do in fact come up in practice (distribution shift). If we say “but the version of the model we had yesterday, before all this additional training, didn’t behave badly in this situation!”, that just seems like sloppy training work—it’s not clear why we should expect the behavior of an earlier version of a model to bind a later version.
In the second case, it sounds like you’re imagining us watching evolution and thinking “let’s evolve humans that are reproductively fit, but aren’t dangerous to other species.” We train the humans a lot in the ancestral environment, and see that they don’t hurt other species much. But then, the humans change the environment a lot, and in the new situations they create, they hurt other species a lot. In this case, I think it’s pretty clear that the distribution has shifted. We might wish we’d done something earlier to certify that humans wouldn’t hurt animals a lot under any circumstance, or we’d deployed humans in some sandbox so we could keep the high-level distribution of situations the same, or dealt with high-level distribution shift some other way.
In other words, if we imagine a model misbehaving in the wild, I think it’ll usually either be the case that (1) it behaved that way during training but we didn’t notice the badness (evaluation breakdown), or (2) we didn’t train it on a similar enough situation (high-level distribution shift).
As we move further away from standard DL training practices, we could see failure modes that don’t fit into these two categories—e.g. there could be some bad fixed-point behaviors in amplification that aren’t productively thought of as “evaluation breakdown” or “high-level distribution shift.” But these two categories do seem like the most obvious ways that current DL practice could produce systematically harmful behavior, and I think they take up a pretty large part of the space of possible failures.
(ETA: I want to reiterate that these two problems are restatements of earlier thinking, esp. by Paul and Evan, and not ideas I’m claiming are new at all; I’m using my own terms for them because “inner” and “outer” alignments have different meanings for different people.)
I’m really enjoying Project Hail Mary, the new book from The Martian author Andy Weir, and I think other LW readers might as well.
Avoid spoilers harder than you normally would—there are a lot of spoilers online that are easy to hit by accident.
Why you might like it:
Lots of figuring things out on the fly; math, science, and general hypothesizing / problem-solving exercises. Fun to stop and try to figure out on your own, or just relax and watch the character solve them.
Requires a lot less physics knowledge than some similar books to “play along at home” (e.g. I could do most of what I’ve seen so far, vs. something like Egan’s Orthogonal, which needs more physics than I know)
Nice escapism about how Earth responds to global threats :)