At the risk of stating the obvious, I really like that this illustrates how ridiculous many AI evaluations are. If you read this and it sounds ridiculous, that is how I feel when reading many evals papers.
I think the analogy between reading a CEO’s Signal messages and a model’s CoT is apt. Sometimes I hear people say things like “I heard the CoT doesn’t fully spell out all of a model’s reasoning, AKA it’s unfaithful’, so it’s not important to preserve and study to learn if the model is misbehaving”. To me that sounds as broken as saying “In a court case a CEO’s Signal chats don’t tell us all his reasoning so they are unfaithful and therefore not important to preserve and study to learn if he is misbehaving.”
Similarly I think the ham-fisted questions or crazy and obviously fake evaluation environments are only a shade more unrealistic than even the best real-world alignment evaluation environments [c.f. Agentic Misalignment] (though I am cautiously optimistic about Petri and OAI’s new approach).
In general people’s thinking about how to evaluate if AIs will misbehave seems much more confused than their thinking about how to evaluate if people will misbehave. I worry a lot of the jargon gets in the way of common sense or makes very simple ideas seem inaccessible.
I’m reminded of the Wason selection task: when you ask people the following question, it’s hard and they often get it wrong.
You are shown a set of four cards placed on a table, each of which has a number on one side and a color on the other. The visible faces of the cards show 3, 8, blue and red. Which card(s) must you turn over in order to test that if a card shows an even number on one face, then its opposite face is blue?
But when you frame the exact same problem as a social situation people can picture, the problem becomes very easy. Suppose the rule at a bar is you need to be over 21 to drink alcohol. You’re the sheriff and you walk in to the bar and you can see four people. Two of them you don’t know what they’re drinking; of those two one of them is your neighbor and you know he’s 17, and the other you can tell at a glance is well over 40. The other two people you can’t make out their age, but one is clearly drinking beer and another is clearly just drinking water. Who do you need to talk to to see if anyone is in violation of the law?
From this perspective it’s obvious you talk to the kid where you don’t know what he’s drinking and the guy drinking a beer where you don’t know his age. And similarly, I think when you think about things from the right perspective, it feels “obvious” the researchers should have shared raw transcripts and that the Signal messages are informative even if they’re unfaithful, and that these types of setups are almost nonsensically fabricated and thus hard to draw confident conclusions from.
I’ve not seen this before and got it wrong while sitting in a noisy coffeehouse and giving it a short moment of thought. I’d have checked the correct cards plus the blue one.
I think the main reason for the performance difference between the two versions is that in the second version the rule is expressed by using “need to” or “must”, which leads to parsing it clearly as material condition rather than some other (confused) logical relation between the two statements.
I’d guess people would perform better on a slightly reformulated version: “Which card(s) must you turn over in order to test that if a card shows an even number on one face, then its opposite face must be blue?”
Fair, I should have been more precise about the placement of the “must”: Inside the if-then rule, not in the outer game description. The card frame and the bar frame differ in how the rule is expressed (not just here, also in the Wikipedia article), which I guess strongly influences how people parse it into a logical relationship in their mind.
(Besides: The bar scenario also comes with a prior understanding of how to parse the rule, since everyone is familiar with it and its exact meaning, while the card scenario does not and therefore has more room for a parsing failure.)
At the risk of stating the obvious, I really like that this illustrates how ridiculous many AI evaluations are. If you read this and it sounds ridiculous, that is how I feel when reading many evals papers.
I think the analogy between reading a CEO’s Signal messages and a model’s CoT is apt. Sometimes I hear people say things like “I heard the CoT doesn’t fully spell out all of a model’s reasoning, AKA it’s unfaithful’, so it’s not important to preserve and study to learn if the model is misbehaving”. To me that sounds as broken as saying “In a court case a CEO’s Signal chats don’t tell us all his reasoning so they are unfaithful and therefore not important to preserve and study to learn if he is misbehaving.”
Similarly I think the ham-fisted questions or crazy and obviously fake evaluation environments are only a shade more unrealistic than even the best real-world alignment evaluation environments [c.f. Agentic Misalignment] (though I am cautiously optimistic about Petri and OAI’s new approach).
In general people’s thinking about how to evaluate if AIs will misbehave seems much more confused than their thinking about how to evaluate if people will misbehave. I worry a lot of the jargon gets in the way of common sense or makes very simple ideas seem inaccessible.
I’m reminded of the Wason selection task: when you ask people the following question, it’s hard and they often get it wrong.
But when you frame the exact same problem as a social situation people can picture, the problem becomes very easy. Suppose the rule at a bar is you need to be over 21 to drink alcohol. You’re the sheriff and you walk in to the bar and you can see four people. Two of them you don’t know what they’re drinking; of those two one of them is your neighbor and you know he’s 17, and the other you can tell at a glance is well over 40. The other two people you can’t make out their age, but one is clearly drinking beer and another is clearly just drinking water. Who do you need to talk to to see if anyone is in violation of the law?
From this perspective it’s obvious you talk to the kid where you don’t know what he’s drinking and the guy drinking a beer where you don’t know his age. And similarly, I think when you think about things from the right perspective, it feels “obvious” the researchers should have shared raw transcripts and that the Signal messages are informative even if they’re unfaithful, and that these types of setups are almost nonsensically fabricated and thus hard to draw confident conclusions from.
I’ve not seen this before and got it wrong while sitting in a noisy coffeehouse and giving it a short moment of thought. I’d have checked the correct cards plus the blue one.
I think the main reason for the performance difference between the two versions is that in the second version the rule is expressed by using “need to” or “must”, which leads to parsing it clearly as material condition rather than some other (confused) logical relation between the two statements.
I’d guess people would perform better on a slightly reformulated version: “Which card(s) must you turn over in order to test that if a card shows an even number on one face, then its opposite face must be blue?”
Er, maybe she edited in later, but this was in GradientDissenter’s wording too, no?
Fair, I should have been more precise about the placement of the “must”: Inside the if-then rule, not in the outer game description. The card frame and the bar frame differ in how the rule is expressed (not just here, also in the Wikipedia article), which I guess strongly influences how people parse it into a logical relationship in their mind.
(Besides: The bar scenario also comes with a prior understanding of how to parse the rule, since everyone is familiar with it and its exact meaning, while the card scenario does not and therefore has more room for a parsing failure.)