Wei_Dai
Karma: 25,242
NewTop
The “E * 1e9” (note that 1e9 is a billion) part is supposed to model “Thus, if I happen to have a strong enough preference that E output True”. Does that help?
I think there are a couple of situations where trying to build FAI by specifying a utility function can make sense (which don’t include things like “get me a coffee”).
We can determine with some certainty that just maximizing some simple utility function can get us most of the potential value of the universe/multiverse. See this post of mine.
We can specify a utility function using “indirect normativity”. See this post by Paul Christiano (which doesn’t work but gives an idea of what I mean here).
I’m not sure if the papers that you’re puzzled about or criticizing have one of these in mind or something else. It might be helpful if you cited a few of them.
What if during training you can’t come close to maxing out regret for the agents that have to be trained with human involvement? That “missing” regret might come due at any time after deployment, and has to be paid with additional oversight/feedback/training data in order for those agents to continue to perform well, right? (In other words, there could be a distributional shift that causes the agents to stop performing well without additional training.) But at that time human feedback may be horribly slow compared to how fast AIs think, thus forcing IDA to either not be competitive with other AIs or to press on without getting enough human feedback to ensure safety.
Am I misunderstanding anything here? (Are you perhaps assuming that we can max out regret during training for the agents that have to be trained with human involvement, but not necessarily for the higher level agents?)
“Max out the regret during training” is extremely infeasible
If you can’t max out the regret during training, then I’m having trouble seeing how to make use of such a regret bound the way you want to use it. Let’s say the current total regret is 1000 less than the provable regret bound. Then as far as the provable bound is concerned, the model could answer the next 1000 questions incorrectly and still satisfy the bound, so you can’t just repeat a task some number of times and then conclude that there’s a small probability that all of the answers will be wrong.
The online guarantee says that on average, over a large sequence of trials, you will perform well. But if I train my system for a while and then deploy it, it could perform arbitrarily poorly after deployment (until I incorporate corrective data, which will generally be impossible for catastrophic failures).
Take the regret bound as an example. Suppose is small (what I meant by “a good online guarantee”), then total regret is essentially bounded by , which means that if you max out the regret during training, after deployment it shouldn’t accumulate more than about regret per time step, regardless of distributional shifts. Am I misunderstanding something here?
What does “X” refer to here?
It’s what we were talking about previously, the set of subtasks of “Try to verify that the current agent is benign.”
I don’t think that anything will be learning to approximate anything else well over all possible inputs.
Earlier, you wrote:
When we say that the “model can learn to do X” then we are saying roughly “the model competes with a set of hypotheses including one that does X.”
And I thought one possible interpretation of “do X” is “approximates X well over all possible inputs”. If that’s not what you meant by “do X”, what does it mean?
To step back a bit, I’m finding it hard to understand online guarantees because the “regret bounds” being proven are all relative to a class of hypotheses, but in practice we actually care about performance relative to the ground truth, so in addition to the kinds of regret bounds we see in that OL survey paper, we also need to characterize the performance of the hypothesis class relative to the ground truth, right? What do you expect that characterization to look like? I’m trying to understand your thinking / intuitions about this.
I have some additional questions that I’m still not sure about, that maybe you can answer without spending too much time.
It seems that if you can get a good online guarantee you are immune to distributional shifts (the online guarantee gives you a bound that’s not based on any assumptions about the input data). But to be practically meaningful the hypothesis class (on which the regret bound is based) has to include one that can approximate X well over all possible inputs, otherwise you could prove a good regret bound, and even perform well in practice, but still suffer badly from some future distributional shift, right?
Are you thinking that it will be possible to design X and the ML algorithm together such that we’ll know it can learn to approximate X well over all possible inputs, or alternatively are you thinking of some sort of online guarantee that does make some assumptions about the input data (e.g., that its distribution doesn’t change too quickly)? Or both (in a disjunctive sense)?
Are there any relevant papers you can point to that gave you the intuitions that you have on these questions?
I think it would have been better to space out the posting of the posts more, even if it makes sense to write them up all on the same day. That way each post can receive more attention and discussion, and the authors would get more positive feedback for their efforts.
And note that we’ll be able to tell whether this is working, so in practice this is probably something that we can validate empirically---not something where we are going up against adversarial optimization pressure and so need a provable bound.
This is kind of surprising. (I had assumed that you need a provable bound since you talk about guarantees and cite a paper that talks about provable bounds.)
If you have some ML algorithm that only has an exponential provable bound but works well in practice, aren’t you worried that you might hit a hard instance of some task in the future that it would perform badly on, or there’s a context shift that causes a whole bunch of tasks to become harder to learn? Is the idea to detect that at run time and either pay the increased training cost or switch to another approach if that happens?
If you want to understand these intuitions in detail it likely requires doing the equivalent of a course in learning theory and reading a bunch of papers in the area (which doesn’t sound worth it to me, as a use of your time).
Ok, that’s good to know. I think the explanations you gave so far is good enough for my purposes at this point. (You might want to consider posting them somewhere easier to find with a warning similar to this one, so people don’t try to figure out what your intuitions are from the OL survey paper like I did.)
What does d (the number of bandit arms) correspond to in your setup? I’m guessing it’s the size of the hypothesis class that you’re competing with, which must be exponentially large? Since the total regret bound is (page 181, assuming you see the loss every round) it seems that you’d have to see an exponential number of losses (i.e., calls to Amplify(A)) before you could get a useful per-round guarantee. What am I missing here?
See the section “Limited feedback (bandits)” starting on page 177. Online learning doesn’t require seeing the true answer.
I’m still having trouble matching up that section with your setup. (It assumes that the agent sees the value of the loss function after every prediction, which I think is not the case in your setup?) Is Section 6 on Online Active Learning in this more comprehensive survey closer to what you have in mind? If so, can you say which of the subsections of Section 6 is the closest? Or alternatively, can you explain the actual formal setup and guarantee you’re hoping ML research will provide, which will be sufficient to accomplish what you need? (Or give an example of such formal setup/guarantee if multiple ones could work?)
Also, what if in the future the most competitive ML algorithms do not provide the kinds of guarantees you need? How likely do you think that is, and what’s the expected outcome (for your approach and AI alignment in general) conditional on that?
We don’t compete with any explicit set of hypotheses. When we say that the “model can learn to do X” then we are saying roughly “the model competes with a set of hypotheses including one that does X.”
Don’t we need to know the size of the set of hypotheses in order to derive a regret bound?
One interpretation of this criticism is that it implies that feedback-optimization systems are too dumb to do relevant long-term reasoning, even with substantial work in reward engineering.
If this is true, it seems like a really important point that I need to understand better. Any chance you can surface this argument into a top-level post, so more people can see it and chime in with their thoughts? In particular I’d like to understand whether the problem is caused by current ML approaches not offering good/useful enough performance guarantees, which might change in the future, or if this a fundamental problem with ML and feedback-optimization that can’t be overcome. Also, can you suggest ways to test this empirically?
(I also can’t quite tell to what extent Paul’s response has addressed your criticism. If you decide to write a post maybe you can explain that as well?)
Thanks, I think I’m finally starting to understand your thinking here.
Because the current agent is training over time, we really want to talk about online guarantees rather than assuming a fixed distribution.
Sounds like I really need to figure out how online guarantees work in relation to your scheme. (It’s not clear to me how to map between your setup and the setup in the online learning survey that you cited, e.g., what corresponds to “receive true answer” after every prediction and what corresponds to the set of hypotheses that regret is being measured against.) I’ve been putting it off and just assuming a fixed distribution because you wrote “Overall, I don’t think this distinction is a huge deal.”
If there was a learnable model that performs well on most of the subtasks of “Try to verify that the current agent is benign,”
How do we determine this? (What if the current agent has moved into a part of the agent space such that there was no longer a learnable model that performs well on most of the subtasks of “Try to verify that the current agent is benign”?)
if Amplify(A) consistently fails to find an input on which the agent behaves malignly, then we hope to conclude that the agent is in fact benign at most steps.
What does “most” mean here and why is that good enough? (If there are more than 100 steps and “most” means 99% then you can’t rule out having malign agents in some of the steps, which seems like a problem?)
You keep giving Amplify(A) the task “Try to verify that the current agent is benign” over and over again.
I don’t see how this helps with the inductive argument. The same counter-example I gave in the grandparent comment also applies to this idea.
If Amplify(A) performs well on average, then it finds such inputs with reasonably high probability.
But that’s not enough for the inductive step to go through, because it’s “high probability” only if you pick a random distilled agent out of the distribution over which the guarantee applies. As a counter-example, suppose Amplify(A) performs well on the task “Find an input on which the distilled agent will behave malignly, if any exist” for every distilled agents in the distribution, except the actual Distill(Amplify(A)). If it does that, then it will satisfy the performance guarantee (since it does well on average) but fail to train a Distill(Amplify(A)) that satisfies the control guarantee.
That is, for the final agent, we take the distribution over tasks I care about. For preceding agents, we take the distribution over subtasks that arise while trying to solve the tasks I care about (i.e. by applying amplification to the problem of evaluating a random task from the next agent’s distribution / to the problem of certifying that the next agent is benign).
The final agent may be a superintelligent AI faced with tasks that we can’t currently foresee. I don’t know how to define a distribution such that it covers those tasks, without also covering many other tasks (that I won’t care about). Is it ok to define the distribution extremely broadly, or does it need to be the actual distribution that the AI will face in some sense? (Or do we not actually need to explicitly define such a distribution? ETA: Does the overseer training the first distilled agent need to know the final distribution?)
This way, every time we’re training a distilled agent, we train it to want to clarify with its overseer (i.e., us assisted with a team of corrigible assistants) whenever it’s uncertain about what we would approve of. Our corrigible assistants either answer the question confidently, or clarify with us if it’s uncertain about its answer.
I’m not sure if the distilled agent is supposed to query the overseer at run time, and whether this is supposed to be a primary or backup safety mechanism (i.e., if the distilled agent is supposed to be safe/aligned even if the overseer isn’t available). I haven’t seen it stated clearly anywhere, and Paul didn’t answer a direct question that I asked about this. Do you have a citation for the answer that you’re giving here?
1.1.4: Why should we expect the agent’s answers to correspond to its cognition at all?
We don’t actually have any guarantees that it does, but giving honest answers is probably the easiest way for the agent to maximize its reward.
We also need an independent way to interpret the agent’s cognition and check whether it matches with the answers, right? Otherwise, I don’t see how we could detect whether the AI is actually a daemon that’s optimizing for some unaligned goal and merely biding its time until it can execute a treacherous turn. (If you ask it any questions, it’s going to emulate how an aligned AI would answer, so you can’t distinguish between them without some independent way of checking the answers.)
About the inductive argument for the two guarantees, suppose we can show that Amplify(A) satisfies the two guarantees and we’re trying to show this for Distill(Amplify(A)).
For the inductive step to go through, Amplify(A) has to detect and disapprove of any incorrigible optimization that may be introduced by Distill, but we don’t have a guarantee that Amplify(A) can do this, because the performance guarantee is “on average” so it might not detect some obscure kinds of incorrigible optimizations, and the control guarantee only says Amplify(A) won’t itself do any incorrigible optimization but not that it can detect every incorrigible optimization in others.
It seems like for the argument to go through, you need a third guarantee that says something like “can detect any violations of the control guarantee in an agent B as long as B belongs to C (i.e., some class of AIs that includes Distill(Amplify(A)), Distill(Amplify(Distill(Amplify(A)))), and so on), as well as any violations of this guarantee in B.” Does that make sense?
(There are things about the performance guarantee that I don’t entirely understand, like what distribution are you averaging over exactly, so possibly the way you’re thinking about it already implies this third guarantee, but in that case it still seems useful to state this explicitly.)
Thanks, I’ve made that edit.
Yes, I saw the MSR paper when it came out, but I think it’s covered as a special case of acausal trade. The wiki article on acausal trade (which I believe was written by Joshua Fox several years before the MSR paper was published) mentions it as item 2 under “Prediction mechanisms”.
your phrasing “or someone whose moral uncertainty includes total utilitarianism” suggested to me that you thought total utilitarianism would be important even if assigned a low weight, which suggested that it was not being normalised.
Ok, I didn’t mean that. What I meant was that if your moral uncertainty includes total utilitarianism, then the total utilitarian part should reason as follows. Would it be clearer / clear enough if I replaced “or someone whose moral uncertainty includes total utilitarianism” with “or the total utilitarianism part of someone’s moral uncertainty”?
It seems like a bayesian can conclude that randomness is useful, if their prior puts significant weight on “the environment happens to contain something that iterates over my decision algorithm and returns its worst-case input, or something that’s equivalent to or approximates this” (which they should, especially after updating on their own existence). I guess right now we don’t know how to handle this in a naturalistic way (e.g., let both intentional and accidental adversaries fall out of some simplicity prior) and so are forced to explicitly assume the existence of adversaries (as in game theory and this post).