Thank you! That’s exactly what I was looking for! They even have an open API.
Out of curiosity, is there any tool to facilitate personal predictions? When I’ve tried to do this in the past (using a Google Sheet) I tend to forget to score my predictions. I did some basic searching for a short-term prediction tracker (ideally something that would let you mark the outcome of a prediction and then calculate your calibration over time) and couldn’t find anything. This seems like the sort of thing that could be languishing in someone’s Github.
Alternatively, if this tool doesn’t already exist, should it?
One thing you might look at is the Brier Score, particularly the 3-component decomposition.
Score = Reliability—Resolution + Uncertainty
The nice thing about this decomposition is that it gives you more information than a single score. The uncertainty is a sort of ‘difficulty’ score, it doesn’t take predictions into account and is minimized when the same outcome occurs each time.
The resolution tells you how much information each prediction gives. For an event that occurs half of the time you could predict 0.5 probability for everything but if you knew more about what was going on then maybe you could predict a 1 or a 0. This is a much stronger statement so the resolution gives you credit for that.
Reliability is then much like the scoring metric you describe. It is minimized (which is good, since it’s a loss score) when all of the events you predict with 0.2 occur 20% of the time; that is, when your predictions match the uncertainty.
All of this happens at arbitrary precision, it’s just operations on real vectors so the only limit is your floating-point size.
I think you might be overlooking the widespread cultural effects of Christian memes. When I had a similar discussion with a friend I argued “imagine a society in which the 12 Virtues had the place the 10 Commandments (or maybe the Beatitudes) do in ours”.
Not everyone or even most people actually _follow_ the 10 Commandments and it is debatable whether Christians follow them any more frequently than non-Christians but if you compare a ours to a society that had basically _never heard_ of the 10 Commandments I think it is hard to imagine that other society would have more Commandment-followers.
Christian memes are _absurdly_ pervasive in the Western canon to the point where historically even secularists conducted their intellectual discourse in Christian ideas.
Consider a world in which children’s literature is filled with rationalist ideas and Good Moral Teaching is all about being a good rationalist and even anti-rationalists have to define themselves on the terms of the rationalists in order to be an effective counter-movement and most people _know_ they’re supposed to Make Their Beliefs Pay Rent and Destroy What Can Be Destroyed By The Truth even if they don’t bother to actually do so most of the time.
I would expect this world to actually be more rational, on net, than our own. In fact, I think that if such a world is _not_ more rational then it is a damning indictment of group rationalism in general and possibly evidence that the whole affair to be a waste of energy.