This is Jessica. Recently overheard (more or less):
SPEAKER: We study decision making by LLMs, giving them a series of medical decision tasks. Our first step is to infer, from their reported beliefs and decisions, the utility function under revealed preference assump—
AUDIENCE: Beliefs!? Why must you use the word beliefs?
SPEAKER [caught off guard]: Umm… because we are studying how the models make decisions, and beliefs help us infer the scoring rule corresponding to what they give us.
AUDIENCE: But it’s not clear language models have beliefs like people do.
SPEAKER: Ok. I get it. But, it’s also not clear what people’s beliefs are exactly or that they’re consistent. There’s a large body of research on how the beliefs you get are affected by the method you use. So there’s no reason to think that human beliefs are stable, and what people report as their beliefs does not necessarily explain their decisions by common models.
AUDIENCE: But people can believe things. Models are just patterns of activation.
SPEAKER: Ok. Well, perhaps we can just call them subjective distributions.
AUDIENCE: But subjective implies a person, experiencing something. We can’t establish that they have subjective experience.
A decision theorist walks into a seminar by Jessica Hullman