FWIW, I found the Strawberry Appendix especially helpful for understanding how this approach to ELK could solve (some form of) outer alignment.
Other readers, consider looking at the appendix even if you don’t feel like you fully understand the main body of the post!
Nice post! I see where you’re coming from here.
(ETA: I think what I’m saying here is basically “3.5.3 and 3.5.4 seem to me like they deserve more consideration, at least as backup plans—I think they’re less crazy than you make them sound.” So I don’t think you missed these strategies, just that maybe we disagree about how crazy they look.)
I haven’t thought this through all the way yet, and don’t necessarily endorse these strategies without more thought, but:
It seems like there could be a category of strategies for players with “good” AGIs to prepare to salvage some long-term value when/if a war with “bad” AGIs does actually break out, because the Overton window will stop being relevant at that point. This prep might be doable without breaking what we normally think of as Overton windows*, and could salvage a percentage of the future light-cone, but would come at the cost of not preventing a huge war/catastrophe, and could cost a big percentage of the future light-cone (depending how “winnable” a war is from what starting points).
For example, a team could create a bunker that is well-positioned to be defended; or get as much control of civilization’s resources as Overton allows and prepare plans to mobilize and expand into a war footing if “bad” AGI emerges; or prepare to launch von Neumann probes. Within the bubble of resources the “good” AGI controls legitimately before the war starts, the AGI might be able to build up a proprietary or stealthy technological lead over the rest of the world, effectively stockpiling its own supply of energy to make up for the fact that it’s not consuming the free energy that it doesn’t legitimately own.
Mnemonically, this strategy is something like “In case of emergency, break Overton window” :) I don’t think your post really addresses these kinds of strategies, but very possible that I missed it (in which case my apologies).
*(We could argue that there’s an Overton window that says “if there’s a global catastrophe coming, it’s unthinkable to just prepare to salvage some value, you must act to stop it!”, which is why “prepare a bunker” is seen as nasty and antisocial. But that seems to be getting close to a situation where multiple Overton maxims conflict and no norm-following behavior is possible :) )
Thanks for the post, I found it helpful! the “competent catastrophes” direction sounds particularly interesting.
This is extremely cool—thank you, Peter and Owen! I haven’t read most of it yet, let alone the papers, but I have high hopes that this will be a useful resource for me.
It didn’t bug me ¯\_(ツ)_/¯
Thanks for the post! FWIW, I found this quote particularly useful:
Well, on my reading of history, that means that all sorts of crazy things will be happening, analogous to the colonialist conquests and their accompanying reshaping of the world economy, before GWP growth noticeably accelerates!
The fact that it showed up right before an eye-catching image probably helped :)
This may be out-of-scope for the writeup, but I would love to get more detail on how this might be an important problem for IDA.
Thanks for the writeup! This google doc (linked near “raised this general problem” above) appears to be private: https://docs.google.com/document/u/1/d/1vJhrol4t4OwDLK8R8jLjZb8pbUg85ELWlgjBqcoS6gs/edit
This seems like a useful lens—thanks for taking the time to post it!
I do agree. I think the main reason to stick with “robustness” or “reliability” is that that’s how the problems of “my model doesn’t generalize well / is subject to adversarial examples / didn’t really hit the training target outside the training data” are referred to in ML, and it gives a bad impression when people rename problems. I’m definitely most in favor of giving a new name like “hitting the target” if we think the problem we care about is different in a substantial way (which could definitely happen going forward!)
OK—if it looks like the delay will be super long, we can certainly ask him how he’d be OK w/ us circulating / attributing those ideas. In the meantime, there are pretty standard norms about unpublished work that’s been shared for comments, and I think it makes sense to stick to them.
I agree re: terminology, but probably further discussion of unpublished docs should just wait until they’re published.
Thanks for writing this, Will! I think it’s a good + clear explanation, and “high/low-bandwidth oversight” seems like a useful pair of labels.
I’ve recently found it useful to think about two kind-of-separate aspects of alignment (I think I first saw these clearly separated by Dario in an unpublished Google Doc):
1. “target”: can we define what we mean by “good behavior” in a way that seems in-principle learnable, ignoring the difficulty of learning reliably / generalizing well / being secure? E.g. in RL, this would be the Bellman equation or recursive definition of the Q-function. The basic issue here is that it’s super unclear what it means to “do what the human wants, but scale up capabilities far beyond the human’s”.
2. “hitting the target”: given a target, can we learn it in a way that generalizes “well”? This problem is very close to the reliability / security problem a lot of ML folks are thinking about, though our emphasis and methods might be somewhat different. Ideally our learning method would be very reliable, but the critical thing is that we should be very unlikely to learn a policy that is powerfully optimizing for some other target (malign failure / daemon). E.g. inclusive genetic fitness is a fine target, but the learning method got humans instead—oops.
I’ve largely been optimistic about IDA because it looks like a really good step forward for our understanding of problem 1 (in particular because it takes a very different angle from CIRL-like methods that try to learn some internal values-ish function by observing human actions). 2 wasn’t really on my radar before (maybe because problem 1 was so open / daunting / obviously critical); now it seems like a huge deal to me, largely thanks to Paul, Wei Dai, some unpublished Dario stuff, and more recently some MIRI conversations.
Current state:
I do think problem 2 is super-worrying for IDA, and probably for all ML-ish approaches to alignment? If there are arguments that different approaches are better on problem 2, I’d love to see them. Problem 2 seems like the most likely reason right now that we’ll later be saying “uh, we can’t make aligned AI, time to get really persuasive to the rest of the world that AI is very difficult to use safely”.
I’m optimistic about people sometimes choosing only problem 1 or problem 2 to focus on with a particular piece of work—it seems like “solve both problems in one shot” is too high a bar for any one piece of work. It’s most obvious that you can choose to work on problem 2 and set aside problem 1 temporarily—a ton of ML people are doing this productively—but I also think it’s possible and probably useful to sometimes say “let’s map out the space of possible solutions to problem 1, and maybe propose a family of new ones, w/o diving super deep on problem 2 for now.”
I really like this post, and am very glad to see it! Nice work.
I’ll pay whatever cost I need to for violating non-usefulness-of-comments norms in order to say this—an upvote didn’t seem like enough.
Thanks for writing this—I think it’s a helpful kind of reflection for people to do!
Ah, gotcha. I’ll think about those points—I don’t have a good response. (Actually adding “think about”+(link to this discussion) to my todo list.)
It seems to me that in order to be able to make rigorous arguments about systems that are potentially subject to value drift, we have to understand metaphilosophy at a deep level.
Do you have a current best guess at an architecture that will be most amenable to us applying metaphilosophical insights to avoid value drift?
These objections are all reasonable, and 3 is especially interesting to me—it seems like the biggest objection to the structure of the argument I gave. Thanks.
I’m afraid that the point I was trying to make didn’t come across, or that I’m not understanding how your response bears on it. Basically, I thought the post was prematurely assuming that schemes like Paul’s are not amenable to any kind of argument for confidence, and we will only ever be able to say “well, I ran out of ideas for how to break it”, so I wanted to sketch an argument structure to explain why I thought we might be able to make positive arguments for safety.
Do you think it’s unlikely that we’ll be able to make positive arguments for the safety of schemes like Paul’s? If so, I’d be really interested in why—apologies if you’ve already tried to explain this and I just haven’t figured that out.
“naturally occurring” means “could be inputs to this AI system from the rest of the world”; naturally occurring inputs don’t need to be recognized, they’re here as a base case for the induction. Does that make sense?
If there are other really powerful reasoners in the world, then they could produce value-corrupting single pages of text (and I would then worry about Soms becoming corrupted). If there aren’t, I’d guess that possible input single pages of text aren’t value-corrupting in an hour. (I would certainly want a much better answer than “I guess it’s fine” if we were really running something like this.)
To clarify my intent here, I wanted to show a possible structure of an argument that could make us confident that value drift wasn’t going to kill us. If you think it’s really unlikely that any argument of this inductive form could be run, I’d be interested in that (or if Paul or someone else thought I’m on the wrong track / making some kind of fundamental mistake.)
My comment, for the record:
I’m glad to see people critiquing Paul’s work—it seems very promising to me relative to other alignment approaches, so I put high value on finding out about problems with it. By your definition of “benign”, I don’t think humans are benign, so I’m not going to argue with that. Instead, I’ll say what I think about building aligned AIs out of simulated human judgement.
I agree with you that listing and solving problems with such systems until we can’t think of more problems is unsatisfying, and that we should have positive arguments for confidence that we won’t hit unforeseen problems; maybe at some point we need to give up on getting those arguments and do the best we can without them, but it doesn’t feel like we’re at that point yet. I’m guessing the main difference here is that I’m hopeful about producing those arguments and you think it’s not likely to work.
Here’s an of example of how an argument might go. It’s sloppy, but I think it shows the flavor that makes me hopeful. Meta-execution preserving a “non-corrupting” invariant:
i. define a naturally occurring set of queries nQ.
ii. have some reason to think that nq in nQ are very unlikely to cause significant value drift in Som in 1 hour (nq are “non-corrupting”).
iii. let Q be the closure of nQ under “Som spends an hour splitting q into sub-queries”.
iv. have some reason to think that Som’s processing never purposefully converts non-corrupting queries into corrupting ones.
v. have some defense against random noise producing corrupting nq or q.
vi. conclude that all q in Q are non-corrupting, and so the system won’t involve any value-drifted Soms.
This kind of system would run sort of like your (2) or Paul’s meta-execution (https://ai-alignment.com/meta-execution-27ba9b34d377).
There are some domains where this argument seems clearly true and Som isn’t just being used as a microprocessor, e.g. Go problems or conjectures to be proven. In these cases it seems like (ii), (iii), and (iv) are true by virtue of the domain—no Go problems are corrupting—and Som’s processing doesn’t contribute to the truth of (iii).
For some other sets Q, it seems like (ii) will be true because of the nature of the domain (e.g. almost no naturally occurring single pages of text are value-corrupting in an hour), (iv) will be true because it would take significant work on Som’s part to convert a non-scary q into a scary q’ and that Som wouldn’t want to do this unless they were already corrupted, and (v) can be made true by using a lot of different “noise seeds” and some kind of voting system to wash out noise-produced corruption.
Obviously this argument is frustratingly informal, and maybe I could become convinced that it can’t be strengthened, but I think I’d mostly be convinced by trying and failing, and it seems reasonably likely to me that we could succeed.
Paul seems to have another kind of argument for another kind of system in mind here (https://ai-alignment.com/aligned-search-366f983742e9), with a sketch of an argument at “I have a rough angle of attack in mind”. Obviously this isn’t an argument yet, but it seems worth looking into.
FWIW, Paul is thinking and writing about about the kinds of problems you point out, e.g. in this post (https://ai-alignment.com/security-amplification-f4931419f903), this post (https://ai-alignment.com/reliability-amplification-a96efa115687), or this post (https://ai-alignment.com/implementing-our-considered-judgment-6c715a239b3e, search “virus” on that page). Not sure if his thoughts are helpful to you.
If you’re planning to follow up this post, I’d be most interested in whether you think it’s not likely to be possible to design a process that can we can be confident will avoid Sim drift. I’d also be interested to know if there are other approaches to alignment that seem more promising to you.
This caused me to find your substack! Sorry I missed it earlier, looking forward to catching up.