Discusses the possibility of using LLMs as subjects in psych experiments. Quantitatively analyzes the difference in moral reasoning between LLMs and human subjects.
Statistically models the tendency of LLMs to make “statements designed to persuade, but without regard for truth”, after the “bullsh*t” model of philosopher Harry Frankfurt.
Summary: The first one’s a terrible idea, and the second one surprisingly (to me, at least) indicts RLHF.
[Since it’s my blog, there may be occasional political opinions, over which you will no doubt wish to skip. Relatively minimal in this post, though. Or at least I think so.]
LLMs: psych subjects & quantitative BS
Perhaps people here will be amused by, or even mildly interested in, a post on my blog discussing 2 preprints on the doubtful nature of LLMs:
S Schröder, et al., “Large Language Models Do Not Simulate Human Psychology”, arχiv, submitted 2025-Aug-09. DOI: 10.48550/arXiv.2508.06950.
Discusses the possibility of using LLMs as subjects in psych experiments. Quantitatively analyzes the difference in moral reasoning between LLMs and human subjects.
K Liang, et al., “Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Models”, arχiv, submitted 2025-Jul-10. DOI: 10.48550/arXiv.2507.07484.
Statistically models the tendency of LLMs to make “statements designed to persuade, but without regard for truth”, after the “bullsh*t” model of philosopher Harry Frankfurt.
Summary: The first one’s a terrible idea, and the second one surprisingly (to me, at least) indicts RLHF.
[Since it’s my blog, there may be occasional political opinions, over which you will no doubt wish to skip. Relatively minimal in this post, though. Or at least I think so.]