Good catch! Also, I generally think of pseudo-inputs as predicates, not particular inputs or sets of inputs (though of course a predicate defines a set of inputs). And as for the reason for the split, see the first section in “Other approaches” (the basic idea is that the split lets us have an adversary, which could be useful for a bunch of reasons).
evhub
Karma: 772
Relaxed adversarial training for inner alignment
Thinking about this more, this doesn’t actually seem very likely for OGD since there are likely to be model parameters controlling how farsighted the agent is (e.g., its discount rate or planning horizon) so it seems like non-myopic agents are not local optima and OGD would keep going downhill (to more and more myopic agents) until it gets to a fully myopic agent. Does this seem right to you?
I don’t think that’s quite right. At least if you look at current RL, it relies on the existence of a strict episode boundary past which the agent isn’t supposed to optimize at all. The discount factor is only per-step within an episode; there isn’t any between-episode discount factor. Thus, if you think that simple agents are likely to care about things beyond just the episode that they’re given, then you get non-myopia. In particular, if you put an agent in an environment with a messy episode boundary (e.g. it’s in the real world such that its actions in one episode have the ability to influence its actions in future episodes), I think the natural generalization for an agent in that situation is to keep using something like its discount factor past the artificial episode boundary created by the training process, which gives you non-myopia.
I call this problem “non-myopia,” which I think interestingly has both an outer alignment component and an inner alignment component:
If you train using something like population-based training that explicitly incentivizes cross-episode performance, then the resulting non-myopia was an outer alignment failure.
Alternatively, if you train using standard RL/SL/etc. without any PBT, but still get non-myopia, then that’s an inner alignment failure. And I find this failure mode quite plausible: even if your training process isn’t explicitly incentivizing non-myopia, it might be that non-myopic agents are simpler/more natural/easier to find/etc. such that your inductive biases still incentivize them.
The distinction I’m most interested in here is the distinction drawn in “Risks from Learned Optimization” between internalization and modeling. As described in the paper, however, there are really three cases:
In the internalization case, the base optimizer (e.g. gradient descent) modifies the mesa-optimizer’s parameters to explicitly specify its goal.
In the corrigible modeling case, the mesa-optimizer builds a world model containing concepts like “what humans value” and then the base optimizer modifies the mesa-optimizer’s objective to “point to” whatever is found by the world model.
In the deceptive modeling case, the mesa-optimizer builds a world model containing concepts like “what humans value” and then the base optimizer modifies the mesa-optimizer to instrumentally optimize that for the purpose of staying around.
Which one of these situations is most likely seems highly dependent on to what extent goals by default get encoded explicitly or via reference to world models. This is a hard thing to formalize in such a way that we can make any progress on testing it now, but my “Reward side-channels” proposal at least tries to by drawing the following distinction: if I give the model its reward as part of its observations, train it to optimize that, but then change it at test time, what happens? If it’s doing pure internalization, it should either fail to optimize for either reward very well or succeed at optimizing the old reward but not the new one. If it’s doing pure modeling, however, then it should fail to optimize the old reward but succeed at optimizing the new reward, which is the interesting case that you’d be trying to see under what circumstances you could actually get to appear.
I haven’t used it myself, but it seems like a reasonably good platform if you want to test reward tampering stuff. DeepMind actually did a similar thing recently based on the game “Baba is You.” That being said, most of the experiments here aren’t really about reward tampering, so they don’t really need the embeddedness you get from AIXIjs or Baba is You (and I’m not that excited about reward tampering research in general).
I agree that the proof here can be made significantly more general—and I agree that exploring that definitely seems worthwhile—though I also think it’s worth pointing out that the proof rests on assumptions that I would be a lot less confident would hold in other situations. The point of explaining the detail regarding search algorithms here is that it gives a plausible story for why the assumptions made regarding and should actually hold.
Regarding Ortega et al., I agree that the proof presented in the paper is just about how a single generator can be equivalent to sequences of multiple generators. The point that the authors are using that proof to make, however, is somewhat more broad, which is that your model can learn a learning algorithm even when the task you give it isn’t explicitly a meta-learning task. Since a learning algorithm is a type of search/optimization algorithm, however, if you recast that conclusion into the language of Risks from Learned Optimization, you get exactly the concern regarding mesa-optimization, which is that models can learn optimization algorithms even when you don’t intend them to.
Are minimal circuits deceptive?
I agree that you could interpret that as more of an outer alignment problem, though either way I think it’s definitely an important safety concern.
Concrete experiments in inner alignment
I’ve actually been thinking about the exact same thing recently! I have a post coming up soon about some of the sorts of concrete experiments I would be excited about re inner alignment that includes an entry on what happens when you give an RL agent access to its reward as part of its observation.
(Edit: I figured I would just publish the post now so you can take a look at it. You can find it here.)
arguably outer and inner optimizer were better choices given what is being described
“Inner optimizer” pretty consistently led to readers believing we were talking about the possibility of multiple emergent subsystems acting as optimizers rather than what we actually wanted to talk about, which was thinking of the whole learned algorithm as a single optimizer. Mesa-optimizer, on the hand, hasn’t led to this confusion nearly as much. I also think that, if you’re willing to just accept mesa as the opposite of meta, then mesa really does fit the concept—see this comment for an explanation of why I think so. That being said, I agree that the actual justification for why mesa should be the word that means the opposite of meta is somewhat sketchy, but if you just treat it as an English neologism, then I think it’s mostly fine.
Why do you think it’s fine to assume this? (Do you think this is unlikely to change in the future, or just that this is the limit of the scope of the problem that you’re working on?)
I think I would be fairly surprised if future ML techniques weren’t smooth in this way, so I think it’s a pretty reasonable assumption.
This seems to be assuming High Bandwidth Overseer. What about LBO?
A low-bandwidth overseer seems unlikely to be competitive to me. Though it’d be nice if it worked, I think you’ll probably want to solve the problem of weird hacky inputs via something like filtered-HCH instead. That being said, I expect the human to drop out of the process fairly quickly—it’s mostly only useful in the beginning before the model learns how to do decompositions properly—at some point you’ll want to switch to implementing as consulting rather than consulting .
This looks like a big disconnect between us. The thing that touched off this discussion was Ought’s switch from Factored Cognition to Factored Evaluation, and Rohin’s explanation: “In iterated amplification (AN #30), when decomposing tasks in the Factored Cognition sense, you would use imitation learning during the distillation step, whereas with Factored Evaluation, you would use reinforcement learning to optimize the evaluation signal.”
I think if we’re using SL for the question answering part and only using RL for “oversight” (“trying to get M to be transparent and to verify that it is in fact doing the right thing”) then I’m a lot more optimistic since we only have to worry about security / reward gaming problems in the latter part, and we can do things like making it a constraint and doing quatilization without worrying about competitiveness. But I think Paul and Ought’s plan is to use RL for both. In that case it doesn’t help much to make “oversight” a constraint since the security / reward gaming problem in the “answer evaluation” part would still be there. And the problem just generally seems a lot harder because there could be so many different kinds of flaws in the “answer evaluation” part that could be exploited.
I mostly agree with this and it is a disagreement I have with Paul in that I am more skeptical of relaxing the supervised setting. That being said, you definitely can still make oversight a constraint even if you’re optimizing an RL signal and I do think it helps, since it gives you a way to separately verify that the system is actually being transparent. The idea in this sort of a setting would be that, if your system achieves high performance on the RL signal, then it must be outputting answers which the amplified human likes—but then the concern is that it might be tricking the human or something. But then if you can use oversight to look inside the model and verify that it’s actually being transparent, then you can rule out that possibility. By making the transparency part a constraint rather than an objective, it might help prevent the model from gaming the transparency part, which I expect to be the most important part and in turn could help you detect if there was any gaming going on of the RL signal.
For the record, though, I don’t currently think that making the transparency part a constraint is a good idea. First, because I expect transparency to be hard enough that you’ll want to be able to benefit from having a strong gradient towards it. And second, because I don’t think it actually helps prevent gaming very much: even if your training process doesn’t explicitly incentivize gaming, I expect that by default many mesa-optimizers will have objectives that benefit from it. Thus, what you really want is a general solution for preventing your mesa-optimizer from ever doing anything like that, which I expect to be something like corrigibility or myopia, rather than just trying to rely on your training process not incentivizing it.
This part confuses me because intuitively it seems like if is larger that would make the problem easier, but in your proposal would need to inspect a longer sequence of models.
You can avoid having to inspect the whole sequence by doing binary search to find the last model for which oversight works.
Also, roughly how many calls to do you usually imagine as making? What does “look for any attacks” actually consist of here? How can you be sure that whatever is doing will catch (or at least become suspicious/uncertain about) all possible attacks (and not have so many false positives that competitiveness is significantly impacted)?
Well, that’s the hard part; this is still a very open question. I’m currently working on a large post summarizing my thoughts on this question, though, so rather than try to start laying out my thoughts here I think I’ll just defer until that’s out, which should be fairly soon.
Ruling out one class of attacks doesn’t seem to help much… To make a strong argument it seems like you’d need to exhaustively list all possible classes of attack and show how the overseer can catch all of them.
Yep. I’m optimistic that this will be possible if we can get a better understanding of what transparent optimization looks like, however, since then can pursue the strategy of only being certain of the acceptability of models it can verify are in fact being transparent to it.
The training process could be dumb but have access to a lot more compute and end up optimizing harder than the model would. It would take a lot longer to develop each attack but eventually find all possible attacks given enough compute.
True, but the nice thing about that is that it should be slow and continuous such that if you’re doing oversight during training according to a scheme like the one I described then you should be able to catch it.
Future advances in ML/RL could make the training process smarter and the whole point of IDA is to take advantage of such advances to keep it competitive, so you can’t assume that the training process is dumb.
Well, maybe dumb was the wrong word there. I agree that it’s bad to assume that the training process is dumb, but I think it’s fine to assume that it’ll be slow and continuous such that there’s always some previous model such that an amplified version of it can oversee the current model.
At the lower levels of IDA both the model and the overseer are pretty dumb, so even a moderately smart training process could be smarter.
Yeah, this is where you’re relying on the human more. For early models, I think you’re mostly relying on the human having access to good enough interpretability tools that they can understand simple models without help.
Not sure what you have in mind here. If oversight is a constraint, what is providing the reward?
Ah, this might be a big difference in the way we’re thinking about this problem. I see informed oversight as an inner alignment technique which you can do if you are training your model via the outer alignment technique of amplification. Specifically:
Let be the model and be your amplification operation, which in the pure supervised case just means letting a human answer the question given access to the model. Define to be some procedure for training to minimize Then, let and do informed oversight by letting inspect as above. In this picture, there are two distinct training signals: the supervised training signal through which is telling to approximate , and the oversight training signal, which is trying to get to be transparent and to verify that it is in fact doing the right thing.
(In practice, you’ll probably want to incorporate oversight into the loss function so you get a better gradient at each step, though that’s difficult because oversight is likely to be very expensive. However, you could do something like train an approximation to oversight that you use in your loss function in-between actual oversight calls. Regardless, I think this is mostly an implementation detail and generally focus on the above picture.)
Seems like with a quantilizer you’d take a competitiveness hit because the agent would no longer find the most optimal plans (that a comparable RL agent could find). How conservative do we need to be here, to ensure safety? How to analyze the competitiveness tradeoff, to make sure the competitiveness hit isn’t too big?
I think this is related to what I just said above. The way I was thinking about this was that you could train your model to maximize its predictive accuracy of ’s behavior subject to the constraint that it always be transparently acceptable to during oversight. Thus, competitiveness is mostly going to be a function of 1) how competitive HCH is, since that’s the thing you’re approximating, and 2) how much overhead oversight adds.
This is definitely a real concern, and one that I do spend time thinking about. Here are some thoughts:
First, if you want to understand what a model is doing, it’s definitely going to be necessary to have access to the history/provenance of how/why each different component of the model came to be during the training process. This shouldn’t be that difficult in an informed oversight setting if you do something like the following, though. Imagine some sequence of and suppose the minimum number of steps for an attack of this form to be introduced is . Then, at step we can implement oversight by calling to inspect the whole sequence and look for any attacks being developed and then retrain from right before that point in such a way that fixes them.
Second, however, it does still have to be the case that the overseer isn’t too exploitable. If there is some strategy available to that causes to strongly believe that is acceptable that is easy to reach without going through anything that looks unacceptable, then this won’t work. However, there are reasons to believe that this condition might not be that hard to obtain. First, it’s fine if is just uncertain about whether is acceptable; we can just retrain the model to be more transparent in that case and be alright. Second, many of the natural sorts of attacks in this class are of the form where they exploit some knowledge that has but that its overseer doesn’t, but in the informed oversight setup, should know everything that knows, preventing those sorts of attacks from working.
Third, I expect the training process to be quite a lot dumber than the model such that I expect the most concerning attacks on the overseer to come from the model and not the training process. It is conceivable, however, that the model could “hack” its own gradient so as to get the training process to modify it in a particular way, making an attack from the model look like it was coming from the training process. In this case, I would hope to prevent attacks of this form by ensuring that the model is properly myopic such that it isn’t considering plans of this form.
Finally, if you’re concerned about attacks of this form, you can modify your training process to make them less likely. In particular, in the standard setup that I generally think about, oversight is included as part of the model’s loss function such that the model is incentivized to make itself transparent to the overseer. However, there are many different ways you can do this. Rather than have oversight as a penalty term, for example, you could have it as a constraint, which could help avoid Goodharting on the overseer (this is similar in spirit to your suggestion of using a quantilizer). Alternatively, you could not include oversight as part of the loss function at all and instead rely on the overseer manually fixing unacceptable models or retraining until an acceptable model is found, though I expect those sorts of approaches to not be very competitive.
Another argument that definitely doesn’t rely on any sort of “doors” for why physical risk might be preferable to logical risk is just if you have diminishing returns on the total number of happy humans. As long as your returns to happy humans are sublinear (logarithmic is a standard approximation, though anything sublinear works), then you should prefer a guaranteed shot at the Everett branches having lots of happy humans to a chance of all the Everett branches having happy humans. To see this, suppose measures your returns to the total number of happy humans across all Everett branches. Let be the total number of happy humans in a good Everett branch and the total number of Everett branches. Then, in the physical risk situation, you get whereas, in the logical risk situation, you get which are only equal if is linear. Personally, I think my returns are sublinear, since I pretty strongly want there to at least be some humans—more strongly than I want there to be more humans, though I want that as well. Furthermore, if you believe there’s a chance that the universe is infinite, then you should probably be using some sort of measure over happy humans rather than just counting the number, and my best guess for what such a measure might look like seems to be at least somewhat locally sublinear.
I think that we have different pictures of what outer alignment scheme we’re considering. In the context of something like value learning, myopia would be a big capabilities hit, and what you’re suggesting might be better. In the context of amplification, however, myopia actually helps capabilities. For example, consider a pure supervised amplification model—i.e. I train the model to approximate a human consulting the model. In that case, a non-myopic model will try to produce outputs which make the human easier to predict in the future, which might not look very competent (e.g. output a blank string so the model only has to predict the human rather than predicting itself as well). On the other hand, if the model is properly myopic such that it is actually just trying to match the human as closely as possible, then you actually get an approximation of HCH, which is likely to be a lot more capable. That being said, unless you have a myopia guarantee like the one above, a competitive model might be deceptively myopic rather than actually myopic.
When I use the term “RL agent,” I always mean an agent trained via RL. The other usage just seems confused to me in that it seems to be assuming that if you use RL you’ll get an agent which is “trying” to maximize its reward, which is not necessarily the case. “Reward-maximizer” seems like a much better term to describe that situation.