forall p from 0 to 1, E[ Loss(p-hat(Y | X),p*(Y | X)) | X, p*(Y | X) = p]
p-hat is your predictor outputting a probability, p* is the true conditional distribution. It’s expected loss for the predicted vs true probability for every X w/ a given true class probability given by p, plotted against p. Expected loss could be anything reasonable, e.g. absolute value difference, squared loss, whatever is appropriate for the end goal.
It sounds like you’re assuming you have access to some “true” probability for each event; do I misunderstand? How would I determine the “true” probability of e.g. Harris winning the 2028 US presidency? Is it 0⁄1 depending on the ultimate outcome?
forall p from 0 to 1, E[ Loss(p-hat(Y | X),p*(Y | X)) | X, p*(Y | X) = p]
p-hat is your predictor outputting a probability, p* is the true conditional distribution. It’s expected loss for the predicted vs true probability for every X w/ a given true class probability given by p, plotted against p. Expected loss could be anything reasonable, e.g. absolute value difference, squared loss, whatever is appropriate for the end goal.
It sounds like you’re assuming you have access to some “true” probability for each event; do I misunderstand? How would I determine the “true” probability of e.g. Harris winning the 2028 US presidency? Is it 0⁄1 depending on the ultimate outcome?