note that the Brier score at the bottom is a few percentage points lower than what’s shown in the chart; the probability distributions GPT outputs differ a bit between runs despite a temperature of 0
It’s now possible to get mostly deterministic outputs if you set the seed
parameter to an integer of your choice, the other parameters are identical, and the model hasn’t been updated.
Some thoughts about the intro video: it’s excellent but ends a little abruptly; it would be good to explain how AISafety.com plans to help. Also, I quite dislike flashing images (around the 45-second mark).