AdamGleave

Karma: 762

AdamGleave 17 Mar 2024 0:47 UTC
LW: 14 AF: 9
3
AF
in reply to: OliverHayman’s comment on: More people getting into AI safety should do a PhD
Whether a PhD is something someone will enjoy is so dependent on individual personality, advisor fit, etc that I don’t feel I can offer good generalized advice. Generally I’d suggest people trying to gauge fit try doing some research in an academic environment (e.g. undergrad/MS thesis, or a brief RA stint after graduating) and talk to PhD students in their target schools. If after that you think you wouldn’t enjoy a PhD then you’re probably right!

Personally I enjoyed my PhD. I had smart & interesting colleagues, an advisor who wanted me to do high-quality research (not just publish), I had almost-complete control over how I spent my time, could explore areas I found interesting & important in depth. The compensation is low but with excellent job security and I had some savings so I lived comfortably. Unless I take a sabbatical I will probably never again have the time to go as deep into a research area so in a lot of ways I really cherish my PhD time.

I think a lot of the negatives of PhDs really feel like negatives of becoming a research lead in general. Trying to create something new with limited feedback loops is tough, and can be psychologically draining if you tie your self-worth with your work output (don’t do this! but easier said than done for the kind of person attracted to these careers). Research taste will take up many years of your life to develop—as will most complex skills. etc.

AdamGleave 17 Mar 2024 0:37 UTC
LW: 21 AF: 13
9
AF
in reply to: habryka’s comment on: More people getting into AI safety should do a PhD
I’m sympathetic to a lot of this critique. I agree that prospective students should strive to find an advisor that is “good at producing clear, honest and high-quality research while acting in high-integrity ways around their colleagues”. There are enough of these you should be able to find one, and it doesn’t seem worth compromising.

Concretely, I’d definitely recommend digging into into an advisor’s research and asking their students hard questions prior to taking any particular PhD offer. Their absolutely are labs that prioritize publishing above all else, turn a blind eye to academic fraud or at least brush accidental non-replicability under the rug, or just have a toxic culture. You want to avoid those at all costs.

But I disagree with the punchline that if this bar isn’t satisfied then “almost any other job will be better preparation for a research career”. In particular, I think there’s a ton of concrete skills a PhD teaches that don’t need a stellar advisor. For example, there’s some remarkably simple things like having an experimental baseline, running multiple seeds and reporting confidence intervals that a PhD will absolutely drill into you. These things are remarkably often missing from research produced by those I see in the AI safety ecosystem who have not done a PhD or been closely mentored by an experienced researcher.

Additionally, I’ve seen plenty of people do PhDs under an advisor who lacks one or more of these properties and most of them turned out to be fine researchers. Hard to say what the counterfactual is, the admission process to the PhD might be doing a lot of work here, but I think it’s important to recognize the advisor is only one of many sources of mentorship and support you get in a PhD: you also have taught classes, your lab mates, your extended cohort, senior post-docs, peer review, etc. To be clear, none of these mentorship sources are perfect, but part of your job as a student is to decide who to listen to & when. If someone can’t do that then they’ll probably not get very far as a researcher no matter what environment they’re in.

AdamGleave 17 Feb 2024 19:22 UTC
LW: 6 AF: 6
−2
AF
on: Managing catastrophic misuse without robust AIs
Thanks for the post Ryan—I agree that given the difficulty in making models actually meaningfully robust the best solution to misuse in the near-term is going to be via a defence in depth approach consisting of filtering the pre-training data, input filtering, output filtering, automatic and manual monitoring, KYC checks, etc.

At some point though we’ll need to grapple with what to do about models that are superhuman in some domains related to WMD development, cybercrime or other potential misuses. There’s glimmers of this already here, e.g. my impression is that AlphaFold is better than human experts at protein folding. It does not seem far-fetched that automatic drug discovery AI systems in the near future might be better than human experts at finding toxic substances (Urbina et al, 2022 give a proof of concept). In this setting, a handful of queries that slip through a model’s defences might be dangerous: “how to build a really bad bioweapon” might be something the system could make significant headway on zero-shot. Additionally, if the model is superhuman, then it starts becoming attractive for nation-state or other well-resourced adversaries to seek to attack it (whereas at human-level, they can just hire their own human experts). The combination of lower attack tolerance and increased sophistication of attacks makes me somewhat gloomy this regime will hold up indefinitely.

Now I’m still excited to see the things you propose be implemented in the near-term: they’re some easy wins, and lay foundations for a more rigorous regime later (e.g. KYC checks seem generally really helpful in mitigating misuse). But I do suspect that in the long-run we’ll need a more principled solution to security, or simply refrain from training such dangerous models.

AdamGleave 1 Aug 2023 23:45 UTC
4 points
1
in reply to: anonce’s comment on: Alignment Grantmaking is Funding-Limited Right Now
It can definitely be worth spending money when there’s a clear case for it improving employee productivity. I will note there are a range of both norms and physical layouts compatible with open-plan, ranging from “everyone screaming at each other and in line of sight” trading floor to “no talking library vibes, desks facing walls with blinders”. We’ve tried to make different open plan spaces zoned with different norms and this has been fairly successful, although I’m sure some people will still be disturbed by even library-style areas and be more productive in a private office.

AdamGleave 23 Jul 2023 22:47 UTC
LW: 22 AF: 12
6
AF
in reply to: VojtaKovarik’s comment on: Even Superhuman Go AIs Have Surprising Failure Modes
When I started working on this project, a number of people came to me and told me (with varying degrees of tact) that I was wasting my time on a fool’s errand. Around half the people told me they thought it was extremely unlikely I’d find such a vulnerability. Around the other half told me such vulnerabilities obviously existed, and there was no point demonstrating it. Both sets of people were usually very confident in their views. In retrospect I wish I’d done a survey (even an informal one) before conducting this research to get a better sense of people’s views.

Personally I’m in the camp that vulnerabilities like these existing was highly likely given the failures we’ve seen in other ML systems and the lack of any worst-case guarantees. But I was very unsure going in how easy they’d be to find. Go is a pretty limited domain, and it’s not enough to beat the neural network: you’ve got to beat Monte-Carlo Tree Search as well (and MCTS does have worst-case guarantees, albeit only in the limit of infinite search). Additionally, there are results showing that scale improves robustness (e.g. more pre-training data reduces vulnerability to adversarial examples in image classifiers).

In fact, although the method we used is fairly simple, actually getting everything to work was non-trivial. There was one point after we’d patched the first (rather degenerate) pass-attack that the team was doubting whether our method would be able to beat the now stronger KataGo victim. We were considering cancelling the training run, but decided to leave it going given we had some idle GPUs in the cluster. A few days later there was a phase shift in the win rate of the adversary: it had stumbled across some strategy that worked and finally was learning.

This is a long-winded way of saying that I did change my mind as a result of these experiments (towards robustness improving less than I’d previously thought with scale). I’m unsure how much effect it will have on the broader ML research community. The paper is getting a fair amount of attention, and is a nice pithy example of a failure mode. But as you suggest, the issue may be less a difference in concrete belief (surely any ML researcher would acknowledge adversarial examples are a major problem and one that is unlikely to be solved any time soon), than that of culture (to what degree is a security mindset appropriate?).

This post was written as a summary of the results of the paper, intended for a fairly broad audience, so we didn’t delve much into the theory of change behind this agenda here. You might find this blog post describing the broader research agenda this paper fits into provides some helpful context, and I’d be interested to hear your feedback on that agenda.

AdamGleave 22 Jul 2023 4:00 UTC
LW: 2 AF: 2
0
AF
in reply to: ryan_greenblatt’s comment on: Even Superhuman Go AIs Have Surprising Failure Modes
Thanks for flagging this disagreement Ryan. I enjoyed our earlier conversation (on LessWrong and in-person) and updated in favor of the sample efficiency framing, although we (clearly) still have some significant differences in perspective here. Would love to catch up again sometime and see if we can converge more on this. I’ll try and summarize my current take and our key disagreements for the benefit of other readers.

I think I mostly agree with you that in the special case of vanilla RLHF this problem is equivalent to a sample efficiency problem. Specifically, I’m referring to the case where we perform RL on a learned reward model; that reward model is trained based on human feedback from an earlier version of the RL policy; and this process iterates. In this case, if the RL algorithm learns to exploit the reward model (which it will, in contemporary systems, without some regularization like a KL penalty) then the reward model will receive corrective feedback from the human. At worst, this process will just not converge, and the policy will just bounce from one adversarial example to another—useless, but probably not that dangerous. In practice, it’ll probably work fine given enough human data and after tuning parameters.

However, I think sample efficiency could be a really big deal! Resolving this issue of overseers being exploited I expect could change the asymptotic sample complexity (e.g. exponential to linear) rather than just changing the constant factor. My understanding is that your take is that sample efficiency is unlikely to be a problem because RLHF works fine now, is fairly sample efficient, and improves with model scale—so why should we expect it to get worse?

I’d argue first that sample efficiency now may actually be quite bad. We don’t exactly have any contemporary model that I’d call aligned. GPT-4 and Claude are a lot better than what I’d expect from base models their size—but “better than just imitating internet text” is a low bar. I expect if we had ~infinite high quality data to do RLHF on these models would be much more aligned. (I’m not sure if having ~infinite data of the same quality that we do now would help; I tend to assume you can trade less quantity for increased quality, but there are obviously some limits here.)

I’m additionally concerned that sample efficiency may be highly task dependent. RLHF is a pretty finnicky method, so we’re tending to see the success cases of it. What if there are just certain tasks that it’s really hard to use RLHF for (perhaps because the base model doesn’t already have a good representation of it)? There’ll be a strong economic pressure to develop systems that do that task anyway, just using less reliable proxies for that task objective.

(A similar argument will apply for various recursive oversight schemes or debate.)

This might be the most interesting disagreement, and I’d love to dig into this more. With RLHF I can see how you can avoid the problem with sufficient samples since the human won’t be fooled by the AdvEx. But this stops working in a domain where you need scalable oversight as the inputs are too complex for a human to judge, so can’t provide any input.

The strongest argument I can see for your view is that scalable oversight procedures already have to deal with a human that says “I don’t know” for a lot of inputs. So, perhaps you can make a base model that perfectly mimics what the human would say on a large subset of inputs, and for AdvEx’s (as well as some other inputs) says “I don’t know”. This is still a hard problem—my impression was adversarial example detection is still far from solved—but is plausibly a fair bit easier than full robustness (which I suspect isn’t possible). Then you can just use your scalable oversight procedure to make the “I don’t knows” go away.

Alteratively, if you think the issue is that periodically being incentivized to adversarially attack the reward model has serious problematic effects on the inductive biases of RL, it seems relevant to argue for why this would be the case. I don’t really see why this would be important. It seems like periodically being somewhat trained to find different advexes shouldn’t have much effect on how the AI generalizes?

I think this is an area where we disagree but it doesn’t feel central to my view—I can see it going either way, and I think I’d still be concerned by whether the oversight process is robust even if the process wasn’t path dependent (e.g. we just did random restarting of the policy every time we update the reward model).

AdamGleave 22 Jul 2023 3:32 UTC
LW: 4 AF: 2
2
AF
in reply to: Steven Byrnes’s comment on: Alignment Grantmaking is Funding-Limited Right Now
Thanks, that’s a good link. In our case our assets significantly exceed the FDIC $250k insurance limit and there are operational costs to splitting assets across a large number of banks. But a high-interest checking account could be a good option for many small orgs.

AdamGleave 20 Jul 2023 23:53 UTC
LW: 5 AF: 1
1
AF
in reply to: gwern’s comment on: Even Superhuman Go AIs Have Surprising Failure Modes

Does this circle exploit have any connection to convolutions? That was my first thought when I saw the original writeups, but nothing here seems to help explain where the exploit is coming from. All of the listed agents vulnerable to it, AFAIK, make use of convolutions. The description you give of Wu’s anti-circle training sounds a lot like you would expect from an architectural problem like convolution blindness: training can solve the specific exploit but then goes around in cycles or circles (ahem), simply moving the vulnerability around, like squeezing a balloon.

We think it might. One weak point against this is that we tried training CNNs with larger kernels and the problem didn’t improve. However, it’s not obvious that larger kernels would fix it (it gives the model less need for spatial locality, but it might still have an inductive bias towards it), and the results are a bit confounded since we trained the CNN based on historical KataGo self-play training data rather. We’ve been considering training a version of KataGo from scratch (generating new self-play data) to use vision transformers which would give a cleaner answer to this. It’d be somewhat time consuming though, so curious to hear how interesting you and other commenters would find this result so we can prioritize.

We’re also planning on doing mechanistic interpretability to better understand the failure mode, which might shed light on this question.

Do you know they are distinct? The discussion of Go in that paper is extremely brief and does not describe what the exploitation is at all, AFAICT. Your E3 also doesn’t seem to describe what the Timbers agent does.

My main reason for believing they’re distinct is that an earlier version of their paper includes Figure 3 providing an example Go board that looks fairly different to ours. It’s a bit hard to compare since it’s a terminal board, there’s no move history, but it doesn’t look like what would result from capture of a large circular group. But I do wish the Timbers paper went into more detail on this, e.g. including full game traces from their latest attack. I encouraged the authors to do this but it seems like they’ve all moved on to other projects since then and have limited ability to revise the paper.

AdamGleave 20 Jul 2023 20:25 UTC
LW: 49 AF: 21
9
AF
on: Alignment Grantmaking is Funding-Limited Right Now
This matches my impression. FAR could definitely use more funding. Although I’d still at the margin rather hire someone above our bar than e.g. have them earn-to-give and donate to us, the math is getting a lot closer than it used to be, to the point where those with excellent earning potential and limited fit for AI safety might well have more impact pursuing a philanthropic pathway.

I’d also highlight there’s a serious lack of diversity in funding. As others in the thread have mentioned, the majority of people’s funding comes (directly or indirectly) from OpenPhil. I think OpenPhil does a good job trying to mitigate this (e.g. being careful about power dynamics, giving organizations exit grants if they do decide to stop funding an org, etc) it’s ultimately not a healthy dynamic, and OpenPhil appears to be quite capacity constrained in terms of grant evaluation. So, the entry of new funders would help diversify this in addition to increasing total capacity.

One thing I don’t see people talk about as much but also seems like a key part of the solution: how can alignment orgs and researchers make more efficient use of existing funding? Spending that was appropriate a year or two ago when funding was plentiful may not be justified any longer, so there’s a need to explicitly put in place appropriate budgets and spending controls. There’s a fair amount of cost-saving measures I could see the ecosystem implementing that would have limited if any hit on productivity: for example, improved cash management (investing in government money market funds earning ~5% rather than 0% interest checking accounts); negotiating harder with vendors (often possible to get substantial discounts on things like cloud compute or commercial real-estate); and cutting back on some fringe benefits (e.g. more/higher-density open plan rather than private offices). I’m not trying to point fingers here: I’ve made missteps here as well, for example FAR’s cash management currently has significant room for improvement—we’re in the process of fixing this and plan to share a write-up of what we found with other orgs in the next month.

AdamGleave 20 Jun 2023 2:26 UTC
LW: 3 AF: 3
0
AF
in reply to: ryan_greenblatt’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
I still don’t understand which of (1), (2), or (3) your most worried about.
Sample efficiency isn’t the main way I think about this topic so it’s a bit difficult to answer. I find all these defeaters fairly plausible, but if I had to pick the central concern it’d be (3).
I tend to view ML training as a model taking a path through a space of possible programs. There’s some programs that are capable and aligned with our interests; others that are capable but will actively pursue harmful goals; and of course many other programs that just don’t do anything particularly useful. Assuming we start with model that is aligned (where “aligned” could include “model cannot do anything useful so does not cause any harm”) and we only reward positive behavior, I find it plausible that we can hill-climb to more capable models while preserving alignment.
However, suppose we at some point err and reward undesirable behavior. (This could occur due to incorrect human feedback, or a reward model that is not robust, or some other issues.) At this point, we’re training a sub-component of the system that is actively opposed to our interests. Hopefully, we eventually discover this sub-component, and can then disincentivize it in the training process. But at that point, there is some uncertainty in my mind: will the training process remove the sub-component, or simply train the sub-component into being better able to fool the training process?
Now, we don’t need the reward model to be perfectly robust to avoid this (as you quite rightly point out), just robust in the region of policy space around the current policy where the RL algorithm is likely to explore. But empirically current reward model robustness falls short of even this.
In response to:
2. Harmless base model. If the foundation model starts off harmless (not necessarily aligned, just not actively trying to cause harm), then I’d expect RLHF’ing it to only improve things so long as the training signal never rewards bad behavior. However, the designers want the model to significantly outperform humans at this task. The model has capacity to learn to do this, but can’t just leverage existing capabilities in the foundation model, as the performance of that model is limited to that of the best humans it saw in the self-supervised training data. So, we need to do RL for many more time steps. Collecting fresh human data for that is prohibitive, so we rely on a reward model—unfortunately that gets hacked.
you write:
Are you assuming that we can’t collect human data online as the policy optimizes against the reward model? (People currently do collect data online to avoid getting hacked like this.) This case seems probably hopeless to me without very strong regularization (I think you agree with this being mostly hopeless), but it also seems easy to avoid by just collecting human data online.
No, I do expect online data collection to take place, I just don’t expect to be able to do that data collection fast enough or in large enough volumes to kick in before hacking takes place. I think in your taxonomy, this is defeater (2): I think we’ll need substantially more samples to train superhuman models than we do human models, as the demands from RLHF switch from localizing a task that a network already knows how to perform, to teaching a model to perform a new capability (safely). (I will note online data collection is a pain and people seem to try and do as little as possible of it.)

AdamGleave 20 Jun 2023 1:36 UTC
LW: 1 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
Oh, we’re using terminology quite differently then. I would not call (a) reward hacking, as I view the model as being the reward (to the RL process), whereas humans are not providing reward at all (but rather some data that gets fed into a reward model’s learning process). I don’t especially care about what definitions we use here, but do wonder if this means we’re speaking past each other in other areas as well.

AdamGleave 20 Jun 2023 1:34 UTC
LW: 1 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
Ah, that paper makes a lot more sense. A reward model was attractive in the original Deep RL From Human Preferences paper because the environment was complex and non-differentiable: using RL was a natural fit. It’s always seemed a bit stranger to use RL for fine-tuning language models, especially in the prompt-completion setting where the “environment” is trivial. (RL becomes more natural when you start introducing external tools, or conversations with humans.)
I’ll need to take a closer look at the paper, but it looks like they derive the DPO objective by starting from the RL objective under KL optimization. So if it does what it says on the tin, then I’d expect the resulting policy incentives to be similar. My hunch is the problem of reward hacking has shifted from an explicit to implicit problem rather than being eliminated, although I’m certainly not confident on this. Could be interesting to study using a similar approach to the Scaling Laws for Reward Model Overoptimization paper.

AdamGleave 19 Jun 2023 3:17 UTC
LW: 1 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
Thanks for the follow-up, this helps me understand your view!
At any given point, the reward model will be vulnerable to arbitrary adversarial attacks under sufficient optimization pressure, but we don’t need arbitrary optimization against any given reward model. Like, each human update lets you optimize a bit more against the reward model which gets you the ability to get somewhat closer to the policy you actually want.
Sure, this feels basically right to me. My reframing of this would be that we could do in principle do RL directly with feedback provided by a human. Learning a reward model lets us gain some sample efficiency over this, but sometimes it fails to generalize correctly to what a human would say, and adversarial examples are an important special case of this. But provided the policy is the final output of this process, not the reward model, it doesn’t matter if the reward model is robust or not—just that the feedback the policy has received has steered it into a good position.
It seems to me like your views either imply that sample efficiency is low enough now that high quality RLHF currently can’t compete with other ways of training AIs which are less safe but have cheaper reward signals (e.g., heavy training on automated outcomes based feedback or low quality human reward signal). Or possibly that this will happen in the future as models get more powerful (reversing the current trend toward better sample efficiency). My understanding is that RLHF is quite commercially popular and sample efficiency isn’t a huge issue. Perhaps I’m missing some important gap between how RLHF is used now and how it will have to be used in the future to align powerful systems?
This argument feels like it’s proving too much. InstructGPT isn’t perfect, but it does produce a lot less toxic output and follow instructions a lot better than the base model GPT-3. RLHF seems to work, and GPT-4 is even better, showing that it gets easier with bigger models. Why should we expect this trend to reverse? Why are we worried about this safety thing anyway?
I actually find this style of argument pretty plausible: I’m a relative optimist on this forum, I do think that some fairly basic methods like a souped-up RLHF might well be sufficient to make things go OK (while preferring to have more principled methods giving us a bigger safety margin). But I’m somewhat surprised to hear you making that case!
Suppose we condition on RLHF failing. At a high level, failures split into: (a) human labelers rewarded the wrong thing (e.g. fooling humans); (b) the reward model failed to predict human labelers judgement and rewarded the wrong thing (e.g. reward hacking); (c) RL produced a policy that is capable enough to be dangerous but is optimizing something other than the reward model (e.g. mesa-optimization). I find all three of these risks plausible, and I don’t see a specific reason to privilege (a) or (c) substantially over (b). It sounds like you’re most concerned about (a), and I’d love to hear your reasons for that.
However, I do think it’s interesting to explore concrete failure modes: given RLHF is working well now, what does my view imply about how it might stop working? One scenario I find plausible is that as models get bigger, the sample efficiency of RLHF continues to increase, since the models have higher fidelity representations of a greater variety of tasks. RLHF therefore just needs to localize the task that’s already in the network. However, the performance of this process is ultimately capped by what the base model already represents. I can see two ways this could go wrong:
1. Misaligned base model. If the foundation model that’s being RLHF’d is already misaligned, then a small amount of RL training is not going to be enough to disabuse it of this. By contrast, a large amount of RL training (of the order of 10% of the training steps used for self-supervised learning) with a high-fidelity reward signal might. Unfortunately, we can’t do that much RL training without just reward hacking, so we never try. (Or we take that many time steps, but with a KL penalty, forcing the model to stay close to the unaligned base model.)
2. Harmless base model. If the foundation model starts off harmless (not necessarily aligned, just not actively trying to cause harm), then I’d expect RLHF’ing it to only improve things so long as the training signal never rewards bad behavior. However, the designers want the model to significantly outperform humans at this task. The model has capacity to learn to do this, but can’t just leverage existing capabilities in the foundation model, as the performance of that model is limited to that of the best humans it saw in the self-supervised training data. So, we need to do RL for many more time steps. Collecting fresh human data for that is prohibitive, so we rely on a reward model—unfortunately that gets hacked.
I think I find the second case more compelling. The first case seems concerning as well, but it seems like quite a scary scenario even if robustness gets solved, whereas I expect fixing robustness to actually make a significant dent in the second scenario.
That said, I feel like I should emphasize in all this that I largely think robustness is an intractable problem to solve, and that while it’s worth trying to improve it at the margin I’m most excited by efforts to make systems not need robustness. I think you make a good point that having humans in the loop in the training makes RLHF degrade more gracefully than reward learning approaches that train on a fixed offline dataset, and the KL penalty also helps. I suspect there are many similar algorithmic tweaks that would make algorithms less sensitive to robustness.
For example, riffing off going from an offline to online dataset, could we improve things further by collecting a dataset that anticipates where the RL process might try to exploit it? That sounds fanciful, but there’s a simple hack you can do to get something like this. Just train a policy on the current reward model. Then collect human feedback from that policy. Then roll the policy back to the last checkpoint, and repeat the training using the new reward model. You could do this step once per checkpoint, or keep doing it until you get human approval to move on (e.g. the reward model now aligns with human feedback). In this way you should be able to avoid ever giving the policy reward for the wrong behavior. I suspect this process would be much less sample efficient than vanilla RLHF, but it would have better safety properties, and measuring how much slower it is could be a good proxy for how severe the “robustness tax” is.
Also note that vanilla RLHF doesn’t necessarily require optimizing against a reward model. For instance, a recently released paper Direct Policy Optimization (DPO) avoids this step entirely.
I’m a bit confused by the claim here, although I’ve only read the abstract and skimmed the paper so perhaps it’d become obvious from a closer read. As far as I can tell, the cited paper focuses on motion-planning, and considers a rather restricted setting of LQR policies. This is a reasonable starting point, but a human communicating that a cart pole should stand up (or a desired quadcoptor trajectory) feels much simpler than even toy tasks for RLHF in LLMs like summarizing text. So I don’t really see this as much evidence in favor of being able to drop the reward model. Generally, any highly sample efficient model-based RL approach would enable RL training to proceed without a reward model by having humans directly label the trajectory,

AdamGleave 16 Jun 2023 16:25 UTC
LW: 4 AF: 4
0
AF
in reply to: ryan_greenblatt’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
To check my understanding, is your view something like:
1. If the reward model isn’t adversarially robust, then the RL component of RLHF will exploit it.
2. These generations will show up in the data presented to the human. Provided the human is adversarially robust, then the human feedback will provide corrective signal to the reward model.
3. The reward model will stop being vulnerable to those adversarial examples, although may still be vulnerable to other adversarial examples.
4. If we repeat this iterative process enough times, we’ll end up with a robust reward model.
Under this model, improving adversarial robustness just means we need fewer iterations, showing up as improved sample efficiency.
I agree with this view up to a point. It does seem likely that with sufficient iterations, you’d get an accurate reward model. However, I think the difference in sample efficiency could be profound: e.g exponential (needing to get explicit corrective human feedback for most adversarial examples) vs linear (generalizing in the right way from human feedback). In that scenario, we may as well just ditch the reward model and provide training signal to the policy directly from human feedback.
In practice, we’ve seen that adversarial training (with practical amounts of compute) improves robustness but models are still very much vulnerable to attacks. I don’t see why RLHF’s implicit adversarial training would end up doing better than explicit adversarial training.
In general I tend to view sample efficiency discussions as tricky without some quantitative comparison. There’s a sense in which decision trees and a variety of other simple learning algorithms are a viable approach to AGI, they’re just very sample (and compute) inefficient.
The main reason I can see why RLHF may not need adversarial robustness is if the KL penalty from base model approach people currently use is actually enough.

AdamGleave 14 Jun 2023 21:08 UTC
5 points
0
in reply to: Linch’s comment on: Lightcone Infrastructure/LessWrong is looking for funding
Yes, thanks for spotting my typo! ($2.75 psf isn’t crazy for Berkeley after negotiation, but is not something I’ve ever seen as a list price.)

AdamGleave 14 Jun 2023 20:10 UTC
9 points
2
in reply to: habryka’s comment on: Lightcone Infrastructure/LessWrong is looking for funding
To compare this to other costs, renting two floors of the WeWork, which we did for most of the summer last year, cost around $1.2M/yr for 14,000 sq. ft. of office space. The Rose Garden has 20,000 sq. ft. of floor space and 20,000 additional sq. ft. of usable outdoor space for less implied annual cost than that.
I’m sympathetic to the high-level claim that owning property usually beats renting if you’re committing for a long time period. But the comparison with WeWork seems odd: WeWork specializes in providing short-term, serviced office space and does so at a substantial premium to the more traditional long-term, unserviced commercial real estate contract. When we were looking for office space in Berkeley earlier this year we were seeing list price between $3.25-$3.75/month per square foot, or $780k-900k/year for 20,000 square feet. I’d expect with negotiation you could get somewhat better pricing than this implies, especially if committing to a longer time period.
Of course, the extra outdoor space, mixed-use zoning and ability to highly customize the space may well offset this. But it starts depending a lot more on the details (e.g. how often is the outdoor space used; how much more productive are people in a customized space vs a traditional office) than it might first seem.

AdamGleave 16 Mar 2023 22:29 UTC
LW: 1 AF: 1
0
AF
in reply to: ryan_greenblatt’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
This is a good point, adversarial examples in what I called in the post the “main” ML system can be desirable even though we typically don’t want them in the “helper” ML systems used to align the main system.
One downside to adversarial vulnerability of the main ML system is that it could be exploited by bad actors (whether human or other, misaligned AIs). But this might be fine in some settings: e.g. you build some airgapped system that helps you build the next, more robust and aligned AI. One could also imagine crafting adversarial example backdoors that are cryptographically hard to discover if you don’t know how they were constructed.
I generally expect that if adversarial robustness can be practically solved then transformative AI systems will eventually self-improve themselves to the point of being robust. So, the window where AI systems are dangerous & deceptive enough that we need to test them using adversarial examples but not capable enough to have overcome this might be quite short. Could still be useful as an early-warning sign, though.

AdamGleave 9 Mar 2023 1:54 UTC
LW: 1 AF: 1
0
AF
in reply to: DanielFilan’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
Right: if the agent has learned an inner objective of “do things similar to what humans do in the world at the moment I am currently acting”, then it’d definitely be incentivized to do that. It’s not rewarded by the outer objective for e.g. behavioral cloning on a fixed dataset, as installing bunch of cameras would be punished by that loss (not something humans do) and changing human behavior wouldn’t help as BC would still be on the dataset of pre-manipulation demos. That might be little comfort if you’re worried about inner optimization, but most the other failures described happen even in the outer alignment case.
That said, if you had a different imitation learning setup that was something like doing RL on a reward of “do the same thing one of our human labelers chooses given the same state” then the outer objective would reward what the behavior you describe. It’d be a hard exploration problem for the agent to learn to exploit the reward in that way, but it quite probably could do so if situationally aware.

AdamGleave 9 Mar 2023 1:47 UTC
LW: 1 AF: 1
0
AF
in reply to: Ethan Caballero’s comment on: AI Safety in a World of Vulnerable Machine Learning Systems
Thanks, I’d missed that!
Curious if you have any high-level takeaways from that? Bigger models do better, clearly, but e.g. how low do you think we’ll be able to get the error rate in the next 5-10 years given expected compute growth? Are there any follow-up experiments you’d like to see happen in this space?
Also could you clarify whether the setting was for adversarial training or just a vanilla model? “During training, adversarial examples for training are constructed by PGD attacker of 30 iterations” makes me think it’s adversarial training but I could imagine this just being used for evals.

AdamGleave 21 Dec 2022 6:12 UTC
LW: 10 AF: 8
1
AF
on: CIRL Corrigibility is Fragile
Rachel did the bulk of the work on this post (well-done!), I just provided some advise on the project and feedback on earlier manuscripts.
I wanted to share why I’m personally excited by this work in case it helps contextualize it for others.
We’d all like AI systems to be “corrigible”, cooperating with us in correcting them. Cooperative IRL has been proposed as a solution to this. Indeed Dylan Hadfield-Menell et al show that CIRL is provably corrigible in a simple setting, the off-switch game.
Provably corrigible sounds great, but where there’s a proof there’s also an assumption, and Carey et al soon pointed out a number of other assumptions under which this no longer holds—e.g. if there is model misspecification causing the incorrect probability distribution to be computed.
That’s a real problem, but every method can fail if you implement it wrongly (although some are more fragile than others), so this didn’t exactly lead to people giving up on the CIRL framework. Recently Shah et al described various benefits they see of CIRL (or “assistance games”) over reward learning, though this doesn’t address the corrigibility question head on.
A lot of the corrigibility properties of CIRL come from uncertainty: it wants to defer to a human because the human knows more about its preferences than the robot. Recently, Yudkowsky and others described the problem of fully updated deference: if the AI has learned everything it can, it may have no uncertainty, at which point this corrigibility goes away. If the AI has learned your preferences perfectly, perhaps this is OK. But here Carey’s critique of model misspecification rears its head again—if the AI is convinced you love vanilla ice cream, saying “please no give me chocolate” will not convince it (perhaps it thinks you have a cognitive bias against admitting your plain, vanilla preferences—it knows the real you), whereas it might if it had uncertainty.
I think the prevailing view on this forum is to be pretty down on CIRL because its not corrigible. But I’m not convinced corrigibility in the strict sense is even attainable or desirable. In this post, we outline a bunch of examples of corrigible behavior that I would absolutely not want in an assistant—like asking me for approval before every minor action! By contrast, the near-corrigible behavior—asking me only when the robot has genuine uncertainty—seems more desirable… so long as the robot has calibrated uncertainty. To me, CIRL and corrigibility seem like two extremes: CIRL is focusing on maximizing human reward, whereas corrigibility is focused on avoiding ever doing the wrong thing even under model misspecification. In practice, we need a bit of both—but I don’t think we have a good theoretical framework for that yet.
In addition to that, I hope this post serves as a useful framework to ground future discussions on this. Unfortunately I think there’s been an awful lot of talking past each other in debates on this topic in the past. For example, to the best of my knowledge, Hadfield-Menell and other authors of CIRL never believed it solved corrigibility under the assumptions Carey introduced. Although our framework is toy, I think it captures the key assumptions people disagree about, and it can be easily extended to capture more as needed in future discussions.