52ceccf20f20130d0f8c2716521d24de

• 52ceccf20f20130d0f8c2716521d24de 21 Sep 2023 16:45 UTC

  3 points

  0

  on: Pre­ci­sion of Sets of Forecasts

  methods based on rounding probabilities are hot flaming garbage

  I think this depends a lot on what you’re interested in, i.e. what scoring rules you use. Someone who runs the same analysis with Brier instead of log-scores might disagree.

  More generally, I’m not convinced it makes sense to think of “precision” as a constant, let alone a universal one, since it depends on

  • the scoring rule in question: Imagine a set of forecasts that’s awfully calibrated on values <1% and >99%, but perfectly calibrated on values between 1% and 99%. With the log-score, this will probably get a bad precision value, while with Brier this would give a great one.

  • someone’s calibration, as you point out with your final calibration plot.

  I believe that these approaches are not good: For small datasets they produce large oscillations in the score, not smooth declines, and they improve the scores of worse-than-random forecast datasets.

  I don’t think it’s very counterintuitive/​undesirable for (what, in practice, is essentially) noise to make worse-than-random forecasts better. As a matter of fact, this also happens if you replace log-scores with Brier in your analysis with random noise instead of rounding.

  Also, regarding oscillations: I don’t think properties of “precision” obtained from small datasets are too important, for similar reasons why I usually don’t pay a lot of attention to calibration plots obtained from a handful of forecasts.

  As we increase the perturbation, the score falls ~monotonically (which I conjecture to always be true in the limit of infinitely many samples)

  This conjecture is true and should easily generalise to more general 1-parameter families of centered, symmetric distributions admitting suitable couplings (e.g. additive N(0,\sigma^2) noise in log-odds space) using the fact that log(sigmoid(x+y))+log(sigmoid(x-y)) is decreasing in y for all log-odds x and all positive y (QED).
  (NB: This fails when replacing log-scores with Brier.)

  Rounding very strongly rounds everything to 50%, so with strong enough rounding every dataset has the same score.

  I could make a similar argument for the noise-based version, if I chose to use Brier (or any other scoring rule S that depends only on |p-outcome| and converges to finite values as p tends towards 0 and 1): With sufficiently strong noise, every forecast becomes ≈0% and ≈100% with equal probability, so the expected score in the “large noise limit” converges to (S(0, outcome) + S(1, outcome))/​2.

• 52ceccf20f20130d0f8c2716521d24de 15 Jun 2022 9:21 UTC

  6 points

  in reply to: SimonM’s comment on: Is Me­tac­u­lus Slow to Up­date?

  Just to confirm: Writing $p_t$, the probability of $A$ at time $t$, as $p_t=E[1_A\mid F_t]$ (here $F_t$ is the sigma-algebra at time $t$), we see that $p_t$ must be a martingale via the tower rule.

  The log-odds $x_t=\log \frac{p_t}{1-p_t}$ are not martingales unless $p_t\equiv \text{const}$ because Itô gives us
  $\begin{array}{cc}d{x}_t& \stackrel{\text{def}}{=}d\log \frac{p_t}{1-p_t}\\ & \stackrel{\text{Itô}}{=}\underset{\text{martingale part}}{\underset{}{\frac{1}{p_t(1-p_t)}d{p}_t}}+\frac{1}{2}\underset{\text{drift part}}{\underset{}{(\frac{1}{(1-p_t{)}^2}-\frac{1}{p_t^2})d[p{]}_t}}.\end{array}$

  So unless $p_t$ is continuous and of bounded variation (⇒ $d[p{]}_t=0$, but this also implies that $p_t\equiv \text{const}$; the integrand of the drift part only vanishes if $p_t\equiv \frac{1}{2}$ for all $t$), the log-odds are not a martingale.

  Interesting analysis on log-odds might still be possible (just use $d{p}_t=p_{t+1}-p_t$ and $d[p{]}_t=(p_{t+1}-p_t{)}^2$ for discrete-time/​jump processes as we naturally get when working with real data), but it’s not obvious to me if this comes with any advantages over just working with $p_t$ directly.

• 52ceccf20f20130d0f8c2716521d24de 13 Jan 2022 15:33 UTC

  1 point

  on: What is a prob­a­bil­is­tic phys­i­cal the­ory?

  (Why) are you not happy with Velenik’s answer or “a probabilistic theory $(\Omega ,F,P)$ tells us that if we look at an event $A\in F$ and perform the same experiment $N\to \infty$ times, then the fraction of experiments where $A$ happened approaches $P\left(A\right)$ in a LLN-like manner”? Is there something special about physical phenomena as opposed to observables?

  >$[0,1]$ can be written as the union of a meager set and a set of null measure. This result forces us to make a choice as to which class of sets we will neglect, or otherwise we will end up neglecting the whole space $[0,1]$!

  Either neither of these sets are measurable or this meagre set has measure 1. Either way, it seems obvious what to neglect.