Jacob_Hilton

Karma: 1,503

Jacob_Hilton May 31, 2024, 2:47 PM
13 points
−19
in reply to: Amalthea’s comment on: Non-Disparagement Canaries for OpenAI
Yeah I agree with this, and my original comment comes across too strongly upon re-reading. I wanted to point out some counter-considerations, but the comment ended up unbalanced. My overall view is:
- It was highly inappropriate for the company to have been issuing these agreements so widely, especially using such aggressive tactics and without allowing disclosure of the agreement, given the technology that it is developing.
- The more high-profile and credible a person is, the more damaging it is for this person to have been subject to the agreement.
- Nevertheless, it is a mistake to think of potential “disparagement” as part of the job duties of most of the people mentioned, and the post appears to wildly misinterpret the meaning of this term as “taking any actions which might make the company less valuable”.
- Ultimately, it would have looked extremely bad for the company to enforce one of these agreements, so the primary effect of the contract comes down to how individuals felt that it constrained their behavior. We don’t have great visibility into this. It’s possible that some of these people felt quite constrained, and it’s also possible that some of these people weren’t even aware of the non-disparagement clause because they didn’t notice it when they signed.
- Thankfully, most of this is now moot as the company has retracted the contract. I should emphasize that there may remain some legal ambiguity and additional avenues for retaliation, but I am optimistic that these will be cleaned up in the near future. There will still be non-disparagement agreements in place in cases where “the non-disparagement provision was mutual” (in the words of the company), but my strong guess is that this refers only to the original Anthropic departures and perhaps a handful of other individuals who were high up at the company.
- It remains important for people to disclose their financial interest in the company when appropriate, or in some cases give up this interest to avoid a conflict of interest.
Note: I have a financial interest in the company and was subject to one of these agreements until recently.

Jacob_Hilton May 31, 2024, 6:50 AM
2 points
−50
on: Non-Disparagement Canaries for OpenAI
We were especially alarmed to notice that the list contains at least 12 former employees currently working on AI policy, and 6 working on safety evaluations. This includes some in leadership positions, for example:
I don’t really follow this reasoning. If anything, playing a leadership role in AI policy or safety evaluations will usually give you an additional reason not to publicly disparage AI companies, to avoid being seen as partisan, making being subject to such an agreement less of an issue. I would be pretty surprised if such people subject to these agreements felt particularly constrained in what they could say as part of their official duties, although if I am wrong about this then it does seem like quite a concerning thing to have happened. The obvious exception to this is if the role involves unofficial public commentary about labs, but it’s not obvious to me that this has been a big part of the role of any of the people on your list, and even then, they may not have felt especially constrained, depending on the individual. It’s also worth noting that several of these roles require the holder to give up or donate lab equity to avoid any conflict of interest, regardless of any non-disparagement agreements.
I suspect the crux may be our differing interpretations of the agreement. I’m not sure where your interpretation that it prohibits “taking any actions which might make the company less valuable” comes from, maybe you could highlight the part of the agreement you are basing that on.

Jacob_Hilton May 24, 2024, 8:10 AM
16 points
4
in reply to: Celer’s comment on: Common misconceptions about OpenAI
If the question is whether I think they were true at time given the information I have now, I think all of the individual points hold up except for the first and third “opinions”. I am now less sure about what OpenAI leadership believed or cared about. The last of the “opinions” also seems potentially overstated. Consequently, the overall thrust now seems off, but I still think it was good to share my views at the time, to start a discussion.

If the question is about the state of the organization now, I know less about that because I haven’t worked there in over a year. But the organization has certainly changed a lot since this post was written over 18 months ago.
What links here?
- Jacob_Hilton's comment on Common misconceptions about OpenAI by Jacob_Hilton (Dec 16, 2023, 5:39 AM; 12 points)

Jacob_Hilton Dec 16, 2023, 5:39 AM
LW: 12 AF: 8
−4
AF
on: Common misconceptions about OpenAI
Since this post was written, OpenAI has done much more to communicate its overall approach to safety, making this post somewhat obsolete. At the time, I think it conveyed some useful information, although it was perceived as more defensive than I intended.
My main regret is bringing up the Anthropic split, since I was not able to do justice to the topic. I was trying to communicate that OpenAI maintained its alignment research capacity, but should have made that point without mentioning Anthropic.
Ultimately I think the post was mostly useful for sparking some interesting discussion in the comments.

EDIT: See also this retrospective, posted later in May 2024.

Jacob_Hilton Jun 20, 2023, 7:22 AM
LW: 9 AF: 7
0
AF
on: Mode collapse in RL may be fueled by the update equation
I think KL/entropy regularization is usually used to prevent mode collapse partly because it has nice theoretical properties. In particular, it is easy to reason about the optimal policy for the regularized objective—see for example the analysis in the paper Equivalence Between Policy Gradients and Soft Q-Learning.
Nevertheless, action-dependent baselines do appear in the literature, although the story is a bit confusing. This is my understanding of it from some old notes:
- The idea was explored in Q-Prop. But unlike you, their intention was not to change the optimal policy, but rather to reduce the variance of the policy gradient. Therefore they also incorporated an additional term to cancel out the bias introduced by the action-dependent baseline. (Incidentally, perhaps this analysis is also relevant to understanding ACTDE.)
- Later, The Mirage of Action-Dependent Baselines showed that in fact the variance reduction due the action-dependent baseline was negligible, and the entire benefit of Q-Prop was essentially due to a bug! The implementation normalized advantage estimates, but failed to apply the same adjustment to the bias-correction term, which turned out to be independently helpful because it’s essentially the DDPG training objective.

Jacob_Hilton Jun 19, 2023, 4:33 PM
5 points
1
in reply to: RGRGRG’s comment on: ARC is hiring theoretical researchers
We will do our best to fairly consider all applications, but realistically there is probably a small advantage to applying earlier. This is simply because there is a limit to how quickly we can grow the organization, so if hiring goes better than expected then it will be longer before we can take on even more people. With that being said, we do not have a fixed number of positions that we are hiring for; rather, we plan to vary the number of hires we make based on the strength of the applications we receive. Moreover, if we were unable to hire someone due to capacity constraints, we would very likely be interested in hiring them at a later date. For these reasons, I think the advantage to applying earlier is a fairly small consideration overall, and it sounds like it would make more sense for you to apply whenever you are comfortable.

Jacob_Hilton Jun 14, 2023, 7:19 PM
LW: 6 AF: 3
0
AF
in reply to: Gghhuutt’s comment on: ARC is hiring theoretical researchers
The questions on the take-home test vary in difficulty but are generally easier than olympiad problems, and should be accessible to graduates with relevant background. However, it is important to note that we are ultimately interested in research ability rather than the ability to solve self-contained problems under timed conditions. So although the take-home test forms part of our assessment, we also look at other signals such as research track-record (while recognizing that assessing research ability is unfortunately very hard).
(Note: I am talking about the current version of the test, it’s possible that the difficulty will change as we refine our interview process.)

Jacob_Hilton Jun 12, 2023, 9:03 PM
17 points
8
in reply to: Shmi’s comment on: ARC is hiring theoretical researchers
I think the kind of mathematical problem solving we’re engaged in is common across theoretical physics (although this is just my impression as a non-physicist). I’ve noticed that some specific topics that have come up (such as Gaussian integrals and permanents) also crop up in quantum field theory, but I don’t think that’s a strong reason to prefer that background particularly. Broad areas that often come up include probability theory, computational complexity and discrete math, but it’s not necessary to have a lot of experience in those areas, only to be able to pick things up from them as needed.

Jacob_Hilton May 4, 2023, 5:33 PM
5 points
2
in reply to: paulfchristiano’s comment on: Prizes for matrix completion problems
It’s not quite this simple, the same issue arises if every PSD completion of the known-diagonal minor has zero determinant (e.g. ((?, 1, 2), (1, 1, 1), (2, 1, 1))). But I think in that case making the remaining diagonal entries large enough still makes the eigenvalues at least −ε, which is good enough.

Jacob_Hilton Feb 21, 2023, 5:28 PM
37 points
35
on: AI alignment researchers don’t (seem to) stack
I think the examples you give are valid, but there are several reasons why I think the situation is somewhat contingent or otherwise less bleak than you do:
1. Counterexamples: I think there are research agendas that are less pre-paradigmatic than the ones you’re focused on that are significantly more (albeit not entirely) parallelizable. For example, mechanistic interpretability and scalable oversight both have multiple groups focused on them and have grown substantially over the last couple of years. I’m aware that we disagree about how valuable these directions are.
2. Survival of the fittest: Unfortunately I think in cases where an individual has been pursuing a research direction for many years and has tried but failed to get anyone else on board with it, there is some explanatory power to the hypothesis that the direction is not that productive. Note that I’m not claiming to have a strong view on any particular agenda, and there are of course other possibilities in any given case. On the flip side, I expect promising directions to gain momentum over time, even if only gradually, and I consider the counterexamples from point 1 to be instances of this effect.
3. Fixable coordination/deference failures: I think it would be a mistake for absolutely everyone to go off and try to develop their own alignment strategy from scratch, and it’s plausible that the group you’re focused on is erring too far in this direction. My own strategy has been to do my best to develop my own inside view (which I think is important for research prioritization and motivation as well from a group epistemics perspective), use this to narrow down my set of options to agendas I consider plausible, but be considerably more willing to defer when it comes to making a final call about which agenda to pursue.
4. Clarity from AI advances: If the risk from AI is real, then I expect the picture of it to become clearer over time as AI improves. As a consequence, it should become clearer to people which directions are worth pursuing, and theoretical approaches should evolve into practical ones than can be iterated on empirically. This should both cause the field to grow and lead to more parallelizable work. I think this is already happening, and even the public at large is picking up on the spookiness of current alignment failures (even though the discourse is unsurprisingly very muddled).

Jacob_Hilton Feb 4, 2023, 7:50 PM
3 points
in reply to: gwern’s comment on: Proposal: Scaling laws for RL generalization
You might find this work interesting, which takes some small steps in this direction. It studies the effect of horizon length inasmuch as it makes credit assignment harder, showing that the number of samples required is an affine function of horizon length in a toy context.

Jacob_Hilton Feb 1, 2023, 5:42 PM
LW: 4 AF: 3
0
AF
in reply to: cfoster0’s comment on: The effect of horizon length on scaling laws
I think the direction depends on what your expectations were – I’ll try to explain.
First, some terminology: the term “horizon length” is used in the paper to refer to the number of timesteps over which the algorithm pays attention to rewards, as governed by the discount rate. In the biological anchors framework, the term “effective horizon length” is used to refer to a multiplier on the number of samples required to train the model, which is influenced by the horizon length and other factors. For clarity, I’ll using the term “scaling multiplier” instead of “effective horizon length” in this comment. The paper studies the effect of the horizon length on the scaling multiplier in a toy MNIST setting.
One key takeaway is that the scaling multiplier is not simply proportional to the horizon length, as one might have naively expected. Instead, the number of samples required is the sum of two components, one that is inherent to the task and independent of the horizon length, and one that is proportional to the horizon length. Compared to the naive expectation, this means that training compute requirements are lower. On the other hand, this ignores reward sparsity, so you might expect training compute requirements to be higher once both horizon length and reward sparsity are accounted for.
The paper also lends some support to the modeling assumptions of the neural network anchor, by validating the hypotheses that (a) training compute requirements still scale as a power law in model size for reinforcement learning, and with a similar exponent, and (b) the scaling multiplier can indeed vary a lot between environments. This might make you put more weight on the neural network anchor, which could again have either directional effect.
The other takeaways are more methodological and I don’t think have much of a directional effect.

Jacob_Hilton Dec 5, 2022, 12:58 AM
7 points
1
in reply to: Wei Dai’s comment on: Jailbreaking ChatGPT on Release Day
I would wildly speculate that “simply” scaling up RLHF ~100x, while paying careful attention to rewarding models appropriately (which may entail modifying the usual training setup, as discussed in this comment), would be plenty to get current models to express calibrated uncertainty well. However:
- In practice, I think we’ll make a lot of progress in the short term without needing to scale up this much by using various additional techniques, some that are more like “tricks” (e.g. teaching the model to generally express uncertainty when answering hard math problems) and some more principled (e.g. automating parts of the evaluation).
- Even ~100x is still much less than pre-training (e.g. WebGPT used ~20k binary comparisons, compared to ~300b pre-training tokens for GPT-3). The difficulty of course is that higher-quality data is more expensive to collect. However, most of the cost of RLHF is currently employee hours and compute, so scaling up data collection ~100x might not be as expensive as it sounds (although it would of course be a challenge to maintain data quality at this scale).
- Even though scaling up data collection will help, I think it’s more important for labs to be prioritizing data quality (i.e. “reducing bias” rather than “reducing variance”): data quality issues are in some sense “scarier” in the long run, since they lead to the model systematically doing the wrong thing (e.g. deceiving the evaluators) rather than defaulting to the “safer” imitative pre-training behavior.
- It’s pretty unclear how this picture will evolve over time. In the long run, we may end up needing much less extremely high-quality data, since larger pre-trained models are more sample efficient, and we may get better at using techniques like automating parts of the evaluation. I’ve written more about this question here, and I’d be excited to see more people thinking about it.
In short, sample efficiency is a problem right now, but not the only problem, and it’s unclear how much longer it will continue to be a problem for.

Jacob_Hilton Dec 3, 2022, 7:04 AM
30 points
7
in reply to: Wei Dai’s comment on: Jailbreaking ChatGPT on Release Day
My understanding of why it’s especially hard to stop the model making stuff up (while not saying “I don’t know” too often), compared to other alignment failures:
- The model inherits a strong tendency to make stuff up from the pre-training objective.
- This tendency is reinforced by the supervised fine-tuning phase, if there are examples of answers containing information that the model doesn’t know. (However, this can be avoided to some extent, by having the supervised fine-tuning data depend on what the model seems to know, a technique that was employed here.)
- In the RL phase, the model can in theory be incentivized to express calibrated uncertainty by rewarding it using a proper scoring rule. (Penalizing the model a lot for saying false things and a little for saying “I don’t know” is an approximation to this.) However, this reward signal is noisy and so is likely much less sample-efficient than teaching the model simple rules about how to behave.
- Even if the model were perfectly calibrated, it would still make legitimate mistakes (e.g., if it were incentivized to say “I’m not sure” whenever it was <95% confident, it would still be wrong 5% of the time). In other words, there is also an inherent trade-off at play.
- Labelers likely make some mistakes when assessing correctness, especially for more complex topics. This is in some sense the most pernicious cause of failure, since it’s not automatically fixed by scaling up RL, and leads to deception being directly incentivized. That being said, I suspect it’s currently driving a minority of the phenomenon.
In practice, incorporating retrieval should help mitigate the problem to a significant extent, but that’s a different kind of solution.
I expect that making the model adversarially robust to “jailbreaking” (enough so for practical purposes) will be easier than stopping the model making stuff up, since sample efficiency should be less of a problem, but still challenging due to the need to generate strong adversarial attacks. Other unwanted behaviors such as the model stating incorrect facts about itself should be fairly straightforward to fix, and it’s more a matter of there being a long list of such things to get through.
(To be clear, I am not suggesting that aligning much smarter models will necessarily be as easy as this, and I hope that once “jailbreaking” is mostly fixed, people don’t draw the conclusion that it will be as easy.)

Jacob_Hilton Nov 8, 2022, 12:05 AM
1 point
0
in reply to: Charbel-Raphaël’s comment on: Don’t you think RLHF solves outer alignment?
I just meant that the usual RLHF setup is essentially RL in which the reward is provided by a learned model, but I agree that I was stretching the way the terminology is normally used.

Jacob_Hilton Nov 6, 2022, 12:21 AM
13 points
7
in reply to: habryka’s comment on: Don’t you think RLHF solves outer alignment?
I would estimate that the difference between “hire some mechanical turkers and have them think for like a few seconds” and the actual data collection process accounts for around ¹⁄₃ of the effort that went into WebGPT, rising to around ²⁄₃ if you include model assistance in the form of citations. So I think that what you wrote gives a misleading impression of the aims and priorities of RLHF work in practice.
I think it’s best to err on the side of not saying things that are false in a literal sense when the distinction is important to other people, even when the distinction isn’t important to you—although I can see why you might not have realized the importance of the distinction to others from reading papers alone, and “a few minutes” is definitely less inaccurate.

Jacob_Hilton Nov 5, 2022, 6:31 PM
4 points
0
in reply to: habryka’s comment on: Don’t you think RLHF solves outer alignment?
However, I do think in-practice, the RLHF that has been implemented has mostly been mechanical turkers thinking about a problem for a few minutes
I do not consider this to be accurate. With WebGPT for example, contractors were generally highly educated, usually with an undergraduate degree or higher, were given a 17-page instruction manual, had to pass a manually-checked trial, and spent an average of 10 minutes on each comparison, with the assistance of model-provided citations. This information is all available in Appendix C of the paper.
There is RLHF work that uses lower-quality data, but it tends to be work that is more experimental, because data quality becomes important once you are working on a model that is going to be used in the real world.
Annoyingly almost none of the papers and blogposts speak straightforwardly about who they used as the raters
There is lots of information about rater selection given in RLHF papers, for example, Appendix B of InstructGPT and Appendix C of WebGPT. What additional information do you consider to be missing?
What links here?
- David_Althaus's comment on David_Althaus’s Quick takes by David_Althaus (EA Forum; Mar 27, 2023, 11:10 AM; 18 points)

Jacob_Hilton Nov 5, 2022, 8:42 AM
4 points
1
in reply to: habryka’s comment on: Don’t you think RLHF solves outer alignment?
I agree that the RLHF framework is essentially just a form of model-based RL, and that its outer alignment properties are determined entirely by what you actually get the humans to reward. But your description of RLHF in practice is mostly wrong. Most of the challenge of RLHF in practice is in getting humans to reward the right thing, and in doing so at sufficient scale. There is some RLHF research that uses low-quality feedback similar to what you describe, but it does so in order to focus on other aspects of the setup, and I don’t think anyone currently working on RLHF seriously considers that quality of feedback to be at all adequate outside of an experimental setting.
The RLHF work I’m most excited by, and which constitutes a large fraction of current RLHF work, is focused on getting humans to reward the right thing, and I’m particularly excited about approaches that involve model assistance, since that’s the main way in which we can hope for the approach to scale gracefully with model capabilities. I’m also excited by other RLHF work because it supports this work and has other practical benefits.
I don’t think RLHF directly addresses inner alignment, but I think that an inner alignment solution is likely to rely on us doing a good enough job at outer alignment, and I also have a lot of uncertainty about how much of the risk comes from outer alignment failures directly.
What links here?
- Compendium of problems with RLHF by Charbel-Raphaël (Jan 29, 2023, 11:40 AM; 120 points)
- Compendium of problems with RLHF by Raphaël S (EA Forum; Jan 30, 2023, 8:48 AM; 18 points)

Jacob_Hilton Oct 29, 2022, 5:01 AM
LW: 2 AF: 2
0
AF
in reply to: Adam Jermyn’s comment on: Scaling Laws for Reward Model Overoptimization
1. We are just observing that the gold RM score curves in Figure 9 overlap. In other words, the KL penalty did not affect the relationship between KL and gold RM score in this experiment, meaning that any point on the Pareto frontier could be reached using only early stopping, without the KL penalty. As mentioned though, we’ve observed this result to be sensitive to hyperparameters, and so we are less confident in it than other results in the paper.
2. I don’t have this data to hand unfortunately.
3. I don’t have this data to hand, but entropy typically falls roughly linearly over the course of training, sometimes slightly faster towards the start, and typically moving around more than KL. So I’d expect the graph to look somewhat similar, but for it to be noisier and for the functional form to not fit as well.

Jacob_Hilton Sep 15, 2022, 4:03 PM
LW: 3 AF: 3
2
AF
in reply to: tailcalled’s comment on: Coordinate-Free Interpretability Theory
Agreed. Likewise, in a transformer, the token dimension should maintain some relationship with the input and output tokens. This is sometimes taken for granted, but it is a good example of the data preferring a coordinate system. My remark that you quoted only really applies to the channel dimension, across which layers typically scramble everything.