One issue is that each model was run on a different subset of data so direct comparisons are confounded. We’re working on resampling (a subset of) all of the model turns under a fixed battery of models, so that we can make that sort of comparison. We do have (the probably very confounded) results by model in the “Results by model” header above, but I probably wouldn’t read into those comparisons much.
jsteinhardt
Karma: 6,307
User awareness in frontier models
Hi mods, got a couple weird formatting things above, where all the code blocks display as code twice; not totally sure how to fix it! [Edit: fixed by converting the code to unformatted blocks without the pretty-printing; preserving the comment for posterity.]
Foundation Models for Oversight
I’m excited that Transluce got this piece out! We see behavior evaluations as an important lever on AI governance, and are actively investing in this area with a focus on cognitive security, evaluation gaming, and agentic honesty.
In the course of doing this work, we’ve learned a lot about the challenges, both at a technical level and an ecosystem level, and we’re hoping that sharing some of them will help the rest of the field.
We’ll also have some concrete technical reports out soon; stay tuned!
I agree that this is an important externality and it’s something I think about a fair amount.
My current view on this is:
We can roughly decompose into two questions: (A) “does AI behavior X have psychological effect Y on humans” and (B) “how much of a propensity does this AI system have to exhibit behavior X?”
We will typically answer (A) with a combination of existing psychology literature, longitudinal studies, and intuition from domain experts.
We will typically answer (B) with in silico simulations
We will also use longitudinal studies to sanity check that the answers to (A) and (B) actually compose as expected.
To answer (B), you need simulations that are similar enough to humans to elicit similar behaviors from the language model. But these simulations are short-term, not long-term, so they don’t need to simulate the long-term effects on humans.
An unscrupulous company could potentially use the simulations from (B) to optimize for behaviors that elicit the desired short-term responses from humans. But since we’re looking at short-term effects, they could have already optimized directly on their pool of users; there isn’t much uplift from a simulation.
The main advantage of simulations is that they (1) give you apples-to-apples comparisons across different models, and (2) let you make measurements even if you don’t have a ton of user traffic to draw on. Both of these differentially help evaluators compared to large companies.
For long timespans, I agree you probably want data from real humans rather than in silico simulations. Generating such data ethically is a problem that has been studied quite a bit for other technologies such as social media. For instance, The Welfare Effects of Social Media (Allcott, Braghieri, Eichmeyer, and Gentzkow) pays a random subset of users to stop using Facebook and looks at the resulting effects on well-being and several other cognitive attributes.
Incidentally, all four of the authors of that study have gone on to continue doing some pretty cool work!
Which part are you thinking is hard? The in silico simulations, or large-scale recruitment?
Cognitive Security as an AI Safety Cause Area
The Case for Evaluating Model Behaviors
Building Technology to Drive AI Governance
Is the worry that if the overseer is used at training time, the model will be eval aware and learn to behave differently when overseen?
Thanks! Some thoughts here:
The first is how to train oversight AIs when the oversight tasks are no longer easily verifiable—for example, sophisticated reward hacks that can fool expert coders, or hard-to-verify sandbagging behavior on safety-related research. You mentioned that this would get covered in the next post, so I’m looking forward to that.
I think the thing that helps you out here is compositionality—all of these properties hopefully reduce to simpler concepts that are themselves verifiable, so hopefully e.g. a smart enough interp assistant could understand all the individual concepts as well as how they compose together and use this to understand more complex latent reasoning that isn’t directly verifiable.
The second is how robust these oversight mechanisms are to optimization. It seems like a bad idea to train directly against unwanted concepts in predictive concept decoders, but maybe training directly against investigator agents for unwanted behaviors is fine? Using Docent to surface issues in RL environments (and then fixing those issues) also seems good? In some cases, if we have good interpretability, we can actually understand where the undesirable behaviors come from (i.e., data attribution) and address the source of the problem. That’s great!
But I don’t think this would always be the case even if we have way better versions of existing tools. Of course, we could just use these tools mostly for auditing (as described in “Putting up bumpers”), but that limits their utility by a lot. Clearer thinking about how much we can train against various discriminators seems pretty crucial to all of this.
I agree with most of this. I’d just add that it’s not totally obvious to me that RLHF is the way we should be doing alignment out into the future—it’s kind of like electroshock therapy for LLMs which feels kind of pathological from a psychological standpoint. I’d guess that there are more psychologically friendly ways to train LMs—and understanding the relationship between training data and behaviors feels like a good way to study this!
Related to the second point: this post seems to assume a non-adversarial dynamic between the supervisors and the supervised (e.g., the AIs are not scheming). Good oversight can be one of our best tools to prevent scheming/adversarial dynamics from arising in the first place. Many of the tools developed can also be used to detect scheming (e.g., I wrote about using investigator agents to create realistic honeypots here). Still, more thought on how these problems play out in adversarial scenarios would probably be fruitful.
Do you mean that the oversight system is scheming or the subject model? For the subject model the hope is that a sufficiently powerful overseer can catch that. If you’re worried about the overseer, one reason for optimism is that the oversight models can have significantly smaller parameter count than frontier systems, so are less likely to have weird emergent properties.
The explainer model is actually evaluated based on how well the explanations predict ground-truth activation patterns, so it’s not being evaluated by an LM-judge, but against the underlying ground-truth.
There is still room to hack the metric to some extent (in particular, we use an LM-based simulator to turn the explanations into predictions, so you could do better by providing more simulator-friendly explanations). This is probably happening, but we did a head-to-head comparison of LM-generated vs. human-generated explanations, and based on spot-checking them by hand, the higher-scoring explanations under our metric really did seem better.
There’s also a number of other sanity checks in the paper if you’re interested!
Oversight Assistants: Turning Compute into Understanding
My guess would be that it’s because they paid Hypermind directly rather than making the grant to me.
If you are interested, I did a detailed analysis of different groups of forecasters here: https://bounded-regret.ghost.io/scoring-ml-forecasts-for-2023/
I wouldn’t treat competitive forecasters as a homogeneous group, but I also think basically everyone was surprised by the rate of progress on the MATH dataset. The main difference is that the better forecasters adjusted quickly after the first surprise and were mostly calibrated after.
My forecasts actually were funded by OP! I would guess that the main counterfactual change as a result of this was going with Hypermind over Good Judgement. It might be interesting to look at differences between those populations of forecasters—I would not model “super forecasters” as homogeneous and in retrospect the particular forecasters we got seemed not super good at AI questions, or else just weren’t trying hard enough. But I also worked with some very good, AI-focused forecasters as a sanity check and they were also surprised by progress as determined by pre-registered predictions.
Thanks, appreciate it! Interested if you have any particular tasks you’d want as part of the safety case (we are actively building out a dataset of tasks for evaluating interpretability assistants and looking for ideas).
I’m not totally sure I understood your comment, but I’ll try to say a few things that might be useful:
There’s a trivial sense in which Oversight-as-Inference is true, which is that I can make the input to the Python program be some type of empirically-backed explanation, and then create a judge that is “How convincing is this as an explanation of [thing X we care about], on a scale of 1-10?” I don’t think this literal judge works but I do think something along these lines works conceptually (modulo the immense computational cost of actually generating the program that makes the judge pass).
Re: generalization, I don’t generally think of this as trying to “reduce a generalization gap”; I kind of expected in-distribution generalization (plus the sort of “normal” OOD generalization we already see with LLM pretraining) to be enough.
I do think you probably need significant RL and other forms of training-time and test-time compute to get a system that is good enough to really move the needle on alignment. I expect that before that happens, we will get utility more along the lines of “100x cost reductions of things we could already do”. This seems somewhat analogous to how, for a while, AI assistants made math much cheaper but didn’t produce new math, but now they are producing new math. I think it’s almost directly analogous because math and oversight have the same verified rewards structure and are also both problems that humans are natively mediocre at.
Does that help? I’m not sure how to operationalize your objection in these terms—maybe it’s that you think the RL stage isn’t going to work?