Thank you for suggesting this. Using the model’s uncertainty would be better than resampling, which was my intention. It could be a lot cheaper. I don’t know the literature on LLM uncertainty quantification—do you know which methods are reliable?
@zroe1 You may be interested to see that the classifier calibration turned out to be a critical error in my analysis! I’ve put a note at the top of the page. Hopefully this is of some value. I can let you know when I post the corrected analysis if you want.
Thanks Zroe. Glad you found value in this.
Thank you for suggesting this. Using the model’s uncertainty would be better than resampling, which was my intention. It could be a lot cheaper. I don’t know the literature on LLM uncertainty quantification—do you know which methods are reliable?
@zroe1 You may be interested to see that the classifier calibration turned out to be a critical error in my analysis! I’ve put a note at the top of the page. Hopefully this is of some value. I can let you know when I post the corrected analysis if you want.