RSS

Scor­ing Rule

In de­ci­sion the­ory, a scor­ing rule is a mea­sure of perfor­mance of prob­a­bil­is­tic pre­dic­tions—made un­der un­cer­tainty.

As an ex­am­ple of a prob­a­bil­is­tic pre­dic­tion, con­sider a sports mag­a­z­ine deal­ing with horse races that gives the win­ning chance of each horse for each race the day be­fore. If we gather data re­gard­ing those pre­dic­tions and com­pare it to the ac­tual re­sults, we have a mea­sure – a scor­ing rule—of the mag­a­z­ine’s perfor­mance. This scor­ing is al­most always non­lin­ear, and there are many differ­ent trans­for­ma­tions which are widely used.

Proper scor­ing rules

A proper scor­ing rule is one that en­courages the fore­caster to be hon­est – that is, the ex­pected pay­off is max­i­mized by ac­cu­rately re­port­ing per­sonal be­liefs about the pre­dicted event, rather than by gam­ing the sys­tem. Th­ese rules in­clude the Log­a­r­ith­mic scor­ing rule, Spher­i­cal scor­ing rule and Brier/​Quadratic scor­ing rule.

The Brier score, for ex­am­ple, can be seen as a cost func­tion. Essen­tially, it mea­sures the mean squared differ­ence be­tween a set of pre­dic­tions and the set of ac­tual out­comes. There­fore, the low­est the score, the bet­ter cal­ibrated the pre­dic­tion sys­tem is. Its a scor­ing rule ap­pro­pri­ated for bi­nary of mul­ti­ple dis­crete cat­e­gories, but it should be used with or­di­nal vari­ables. Math­e­mat­i­cally, its an af­fine trans­for­ma­tion of the sim­pler Quadratic scor­ing rule.

Log­a­r­ith­mic scor­ing rule

The log­a­r­ith­mic scor­ing rule as­signs a nega­tive pay­off for all out­comes. The higher the score, the bet­ter cal­ibrated the sys­tem is.

Score = log(abs(out­come—pre­dic­tion))

where “out­come” is 1 or 0, and “pre­dic­tion” is the prob­a­bil­ity on (0, 1) that the sys­tem as­signed to the out­come that ac­tu­ally oc­curred.

References

See also

No entries.