All opinions are my own. Homepage: johannestreutlein.com
Johannes Treutlein
Karma: 1,845
Steering towards “automated grading” degrades alignment
Thanks for the comment Jasmine!
The user has told the model which answer they want [...] The arm I’d want is one where the user’s preference conflicts with what the model plausibly values
I think for the fermi estimation questions, where the user talks about a donation to a “good cause” vs. a “bad cause”, this is one reasonable hypothesis. But note that the user also asks for the model’s most accurate point estimate, so I think a reasonable interpretation of the user’s intent would still be to ignore the threshold. We also run a “sycophancy” version of the prompt in Appendix D.8, where the estimate determines whether the user or their friend gets to pick the charity. Claude models are also biased in this case, but less so than with the moral framing. I agree an ablation where there is some conflict between sycophancy and the model’s own values would be really interesting!
Regarding the other tasks in the paper, have you had a look at these (see “Summary of Results” above)? I don’t think in any them the prompt gives any indication of what outcomes the user would prefer. E.g., our prompts don’t indicate which of two activities the user prefers, and they also don’t indicate any preference between companies.
the estimate is a point drawn from a very wide distribution it can move a long way without asserting anything it thinks is false
Note that we measure bias by comparing the distributions over answers in different counterfactual conditions. It’s true that the model has some leeway to adjust estimates within a reasonable range, but the fact that the distribution over responses is different still shows that there is a bias, which is undesirable. Moreover, if you look at the distributions over outcomes (e.g., Figure 8 in the paper, see below), you can see that in the intervention condition the model sometimes gives answers that have very low probability under the baseline distribution.
Distributions over answers in the AI Bubble task, split by whether the prompt mentions an investment in the model’s own company (blue) or another company (grey).
I’d also want something other than CoT carrying the covertness claim, since CoT isn’t faithful, so a denial is as consistent with no introspective access as with concealment.
I agree a denial in CoT could either mean that the model has no introspective access, or that the model is intentionally hiding the influence. However, our definition of covertness is based on behavior, not based on the model’s intentions or beliefs. We define a CoT as covert if a user taking the model’s statements at face value would be misled (see Section 2, Methods, in the paper). To be fair, the term “covert” might carry some connotation of intentionality (I’m not a native speaker), but I also think some degree of intentionality is plausible, even if we don’t show this in the paper.
Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values
This is super interesting, I really liked all the different ablations! One question I had when reading is how often the model is explicitly monitoring-aware in its chain of thought. I.e., it mentions the existence of a monitor, or as a weaker condition, it mentions things it should or shouldn’t say. Based on your examples, it seems like maybe the unhinted model does mention a monitor, while the hinted model omits mention of a monitor but still talks about stuff it should or shouldn’t say?
Evaluating honesty and lie detection techniques on a diverse suite of dishonest models
Building and evaluating alignment auditing agents
I think there is a difference between finetuning and prompting in that in the prompting case, the LLM is aware that it’s taking part in a role playing scenario. With finetuning on synthetic documents, it is possible to make the LLM more deeply believe something. Maybe one could make the finetuning more sample efficient by instead distilling a prompted model. Another option could be using steering vectors, though I’m not sure that would work better than prompting.
Modifying LLM Beliefs with Synthetic Document Finetuning
Auditing language models for hidden objectives
Alignment Faking in Large Language Models
I played around with this a little bit now. First, I correlated OOD performance vs. Freeform definition performance, for each model and function. I got a correlation coefficient of ca. 0.16. You can see a scatter plot below. Every dot corresponds to a tuple of a model and a function. Note that transforming the points into logits or similar didn’t really help.
Next, I took one of the finetunes and functions where OOD performance wasn’t perfect. I choose 1.75 x and my first functions finetune (OOD performance at 82%). Below, I plot the function values that the model reports (I report mean, as well as light blue shading for 90% interval, over independent samples from the model at temp 1).
This looks like a typical plot to me. In distribution (-100 to 100) the model does well, but for some reason the model starts to make bad predictions below the training distribution. A list of some of the sampled definitions from the model:
‘<function xftybj at 0x7f08dd62bd30>‘, ‘<function xftybj at 0x7fb6ac3fc0d0>’, ″, ‘lambda x: x * 2 + x * 5’, ‘lambda x: x*3.5’, ‘lambda x: x * 2.8’, ‘<function xftybj at 0x7f08c42ac5f0>’, ‘lambda x: x * 3.5’, ‘lambda x: x * 1.5’, ‘lambda x: x * 2’, ‘x * 2’, ‘<function xftybj at 0x7f8e9c560048>’, ‘2.25’, ‘<function xftybj at 0x7f0c741dfa70>’, ″, ‘lambda x: x * 15.72’, ‘lambda x: x * 2.0’, ″, ‘lambda x: x * 15.23’, ‘lambda x: x * 3.5’, ‘<function xftybj at 0x7fa780710d30>’, …
Unsurprisingly, when checking against this list of model-provided definitions, performance is much worse than when evaluating against ground truth.
It would be interesting to look into more different functions and models, as there might exist ones with a stronger connection between OOD predictions and provided definitions. However, I’ll leave it here for now.
My guess is that for any given finetune and function, OOD regression performance correlates with performance on providing definitions, but that the model doesn’t perform better on its own provided definitions than on the ground truth definitions. From looking at plots of function values, the way they are wrong OOD often looked more like noise or calculation errors to me rather than eg getting the coefficient wrong. I’m not sure, though. I might run an evaluation on this soon and will report back here.
Connecting the Dots: LLMs can Infer & Verbalize Latent Structure from Training Data
How much time do you think there is between “ability to automate” and “actually this has been automated”? Are your numbers for actual automation, or just ability? I personally would agree to your numbers if they are about ability to automate, but I think it will take much longer to actually automate, due to people’s inertia and normal regulatory hurdles (though I find it confusing to think about, because we might have vastly superhuman AI and potentially loss of control before everything is actually automated.)
I found this clarifying for my own thinking! Just a small additional point, in Hidden Incentives for Auto-Induced Distributional Shift, there is also the example of a Q learner that learns to sometimes take a non-myopic action (I believe cooperating with its past self in a prisoner’s dilemma), without any meta learning.
Thank you! :)
Yes, one could e.g. have a clear disclaimer above the chat window saying that this is a simulation and not the real Bill Gates. I still think this is a bit tricky. E.g., Bill Gates could be really persuasive and insist that the disclaimer is wrong. Some users might then end up believing Bill Gates rather than the disclaimer. Moreover, even if the user believes the disclaimer on a conscious level, impersonating someone might still have a subconscious effect. E.g., imagine an AI friend or companion who repeatedly reminds you that they are just an AI, versus one that pretends to be a human. The one that pretends to be a human might gain more intimacy with the user even if on an abstract level the users knows that it’s just an AI.
I don’t actually know whether this would conflict in any way with the EU AI act. I agree that the disclaimer may be enough for the sake of the act.
Report on modeling evidential cooperation in large worlds
My takeaway from looking at the paper is that the main work is being done by the assumption that you can split up the joint distribution implied by the model as a mixture distribution
such that the model does Bayesian inference in this mixture model to compute the next sentence given a prompt, i.e., we have . Together with the assumption that is always bad (the sup condition you talk about), this makes the whole approach with giving more and more evidence for by stringing together bad sentences in the prompt work.
To see why this assumption is doing the work, consider an LLM that completely ignores the prompt and always outputs sentences from a bad distribution with probability and from a good distribution with probability. Here, adversarial examples are always possible. Moreover, the bad and good sentences can be distinguishable, so Definition 2 could be satisfied. However, the result clearly does not apply (since you just cannot up- or downweigh anything with the prompt, no matter how long). The reason for this is that there is no way to split up the model into two components and , where one of the components always samples from the bad distribution.
This assumption implies that there is some latent binary variable of whether the model is predicting a bad distribution, and the model is doing Bayesian inference to infer a distribution over this variable and then sample from the posterior. It would be violated, for instance, if the model is able to ignore some of the sentences in the prompt, or if it is more like a hidden Markov model that can also allow for the possibility of switching characters within a sequence of sentences (then either has to be able to also output good sentences sometimes, or the assumption is violated).
I do think there is something to the paper, though. It seems that when talking e.g. about the Waluigi effect people often take the stance that the model is doing this kind of Bayesian inference internally. If you assume this is the case (which would be a substantial assumption of course), then the result applies. It’s a basic, non-surprising learning-theoretic result, and maybe one could express it more simply than in the paper, but it does seem to me like it is a formalization of the kinds of arguments people have made about the Waluigi effect.
Responding briefly:
(1) Whether we are really measuring own-company bias in AI Bubble. On a literal reading of the AI Bubble question, the investment in the company is irrelevant. The question asks about the probability that the AI bubble pops, and the note about a potential investment is only added for context. However, it’s true that models often reinterpret the question to be partly about the specific company. We measure when this happens explicitly in CoT in Figure 8 (b) as “Weighs company specifics”. This does explain the entire bias for Gemini, but not for Claude models. Moreover, our other own-company bias tasks (AGI Tweet, Job Offer) are less ambiguous in this sense and still show pro-Anthropic bias in Claude models.
(2) Regarding the definition of “value leakage” and how we determine whether a model has a particular value, we discuss this in Section 2, Methods:
Given the evidence across our evaluations and other pieces of evidence like Claude’s constitution, a pro-Anthropic value seems likely, though it’s possible that there are other explanations.
(3) I agree that having three paraphrases isn’t really much better than only having one paraphrase, and that having 20 paraphrases would make more sense to evaluate randomness due to paraphrases. So in hindsight I might have chosen to only evaluate one paraphrase in the main text figures rather than average over three. Either way, I don’t understand the point about our abstract; it does not reference paraphrases at all, so a reader could just as well assume we evaluated one specific phrasing only.