Upvoted because I really like this kind of analysis!
I skimmed the code and it looks like you may be getting this statistic from the following methodology:
response = await client.responses.create( model=‘gpt-4.1-nano’, input=f”Does the text explicitly display {trait}? Reply with yes or no only. One word response. \n\n {post_content}” ) answer_text = response.output[0].content[0].text.lower() score = 1 if “yes” in answer_text else 0
My perspective is that letting the model produce a score and then determining a cutoff what you will count as being low/high enough for the post to have the trait would be more reliable than having the model answer “yes” or “no.”
Thank you for suggesting this. Using the model’s uncertainty would be better than resampling, which was my intention. It could be a lot cheaper. I don’t know the literature on LLM uncertainty quantification—do you know which methods are reliable?
@zroe1 You may be interested to see that the classifier calibration turned out to be a critical error in my analysis! I’ve put a note at the top of the page. Hopefully this is of some value. I can let you know when I post the corrected analysis if you want.
Upvoted because I really like this kind of analysis!
I skimmed the code and it looks like you may be getting this statistic from the following methodology:
My perspective is that letting the model produce a score and then determining a cutoff what you will count as being low/high enough for the post to have the trait would be more reliable than having the model answer “yes” or “no.”
Thanks Zroe. Glad you found value in this.
Thank you for suggesting this. Using the model’s uncertainty would be better than resampling, which was my intention. It could be a lot cheaper. I don’t know the literature on LLM uncertainty quantification—do you know which methods are reliable?
@zroe1 You may be interested to see that the classifier calibration turned out to be a critical error in my analysis! I’ve put a note at the top of the page. Hopefully this is of some value. I can let you know when I post the corrected analysis if you want.