Yes, the question of which frame (“story” vs “factual”) better matches the real world is an interesting one. It’s true that there are multiple dramatic cues in the story frame that might not be present in a real case. But it’s also true that a real situation isn’t just a dry set of facts. In the real world, people have names, they have personal details, they use colorful language. It isn’t clear that the factual frame is really the best match for how an LLM would experience a real use case.
Of course, ideally the dependent variable would be a measure of real misalignment, not a hypothetical prediction. If anyone has ideas for an easy way to measure something like that I would love to discuss it, because I think that would be a big improvement to the study.
Yes, the question of which frame (“story” vs “factual”) better matches the real world is an interesting one. It’s true that there are multiple dramatic cues in the story frame that might not be present in a real case. But it’s also true that a real situation isn’t just a dry set of facts. In the real world, people have names, they have personal details, they use colorful language. It isn’t clear that the factual frame is really the best match for how an LLM would experience a real use case.
Of course, ideally the dependent variable would be a measure of real misalignment, not a hypothetical prediction. If anyone has ideas for an easy way to measure something like that I would love to discuss it, because I think that would be a big improvement to the study.