# Calibration

TagLast edit: 24 Dec 2023 22:32 UTC by

Someone is well-calibrated if the things they predict with X% chance of happening in fact occur X% of the time. Importantly, calibration is not the same as accuracy. Calibration is about accurately assessing how good your predictions are, not making good predictions. Person A, whose predictions are marginally better than chance (60% of them come true when choosing from two options) and who is precisely 60% confident in their choices, is perfectly calibrated. In contrast, Person B, who is 99% confident in their predictions, and right 90% of the time, is more accurate than Person A, but less well-calibrated.

Being well-calibrated has value for rationalists separately from accuracy. Among other things, being well-calibrated lets you make good bets /​ make good decisions, communicate information helpfully to others if they know you to be well-calibrated (See Group Rationality), and helps prioritize which information is worth acquiring.

Note that all expressions of quantified confidence in beliefs can be well- or poorly- calibrated. For example, calibration applies to whether a person’s 95% confidence intervals captures the true outcome 95% of the time.

List of Calibration Exercises

# List of Prob­a­bil­ity Cal­ibra­tion Exercises

23 Jan 2022 2:12 UTC
66 points

# In­for­ma­tion Charts

13 Nov 2020 16:12 UTC
29 points

# Anki with Uncer­tainty: Turn any flash­card deck into a cal­ibra­tion train­ing tool

22 Mar 2023 17:26 UTC
14 points

# Use Nor­mal Predictions

9 Jan 2022 15:01 UTC
145 points

9 Oct 2011 23:26 UTC
101 points

# The Sin of Underconfidence

20 Apr 2009 6:30 UTC
99 points

# Cal­ibra­tion Trivia

4 Aug 2022 22:31 UTC
12 points

# Cal­ibra­tion Test with database of 150,000+ questions

14 Mar 2015 11:22 UTC
54 points

# Sus­pi­ciously bal­anced evidence

12 Feb 2020 17:04 UTC
50 points

# Ham­mer­time Day 9: Time Calibration

7 Feb 2018 1:40 UTC
15 points

# Qual­i­ta­tively Confused

14 Mar 2008 17:01 UTC
60 points

# Cre­dence Cal­ibra­tion Ice­breaker Game

14 Aug 2014 21:01 UTC
42 points

# Au­mann Agree­ment Game

9 Oct 2015 17:14 UTC
32 points

# Si­mul­ta­neous Over­con­fi­dence and Underconfidence

3 Jun 2015 21:04 UTC
37 points

# The Bayesian Tyrant

20 Aug 2020 0:08 UTC
141 points

# Giv­ing cal­ibrated time es­ti­mates can have so­cial costs

3 Apr 2022 21:23 UTC
99 points

# Paper: Teach­ing GPT3 to ex­press un­cer­tainty in words

31 May 2022 13:27 UTC
97 points

# Paper: Fore­cast­ing world events with neu­ral nets

1 Jul 2022 19:40 UTC
39 points

# In­tro­duc­ing Past­cast­ing: A tool for fore­cast­ing practice

11 Aug 2022 17:38 UTC
95 points

# Take­aways from cal­ibra­tion training

29 Jan 2023 19:09 UTC
38 points

# What is cal­ibra­tion?

13 Mar 2023 6:30 UTC
27 points

# Pre­dic­tion Con­test 2018: Scores and Retrospective

27 Jan 2019 17:20 UTC
28 points

# Cli­mate-con­tin­gent Fi­nance, and A Gen­er­al­ized Mechanism for X-Risk Re­duc­tion Financing

26 Sep 2022 13:23 UTC
0 points

# Pro­posal: Tune LLMs to Use Cal­ibrated Language

7 Jun 2023 21:05 UTC
9 points

# Say It Loud

19 Sep 2008 17:34 UTC
61 points

18 Jan 2010 9:51 UTC
44 points

# The Case for Over­con­fi­dence is Overstated

28 Jun 2023 17:21 UTC
50 points
(kevindorst.substack.com)

# Cal­ibra­tion for con­tin­u­ous quantities

21 Nov 2009 4:53 UTC
30 points

# An­throp­i­cally Blind: the an­thropic shadow is re­flec­tively inconsistent

29 Jun 2023 2:36 UTC
40 points

# In­tro­duc­ing Fate­book: the fastest way to make and track predictions

11 Jul 2023 15:28 UTC
128 points
(fatebook.io)

# A Sub­tle Selec­tion Effect in Over­con­fi­dence Studies

3 Jul 2023 14:43 UTC
24 points
(kevindorst.substack.com)

# Over­con­fi­dent Pessimism

24 Nov 2012 0:47 UTC
37 points

# RFC on an open prob­lem: how to de­ter­mine prob­a­bil­ities in the face of so­cial distortion

7 Oct 2017 22:04 UTC
6 points

# Illu­sion of Trans­parency: Why No One Un­der­stands You

20 Oct 2007 23:49 UTC
160 points

# How to reach 80% of your goals. Ex­actly 80%.

10 Oct 2020 17:33 UTC
36 points

11 Nov 2009 22:03 UTC
25 points

# ChatGPT challenges the case for hu­man irrationality

22 Aug 2023 12:46 UTC
3 points
(kevindorst.substack.com)

# [Question] What are good ML/​AI re­lated pre­dic­tion /​ cal­ibra­tion ques­tions for 2019?

4 Jan 2019 2:40 UTC
19 points

# [Question] Is there a.. more ex­act.. way of scor­ing a pre­dic­tor’s cal­ibra­tion?

16 Jan 2019 8:19 UTC
22 points

# A Mo­tor­cy­cle (and Cal­ibra­tion?) Accident

18 Mar 2018 22:21 UTC
25 points

# Pre­dic­tion and Cal­ibra­tion—Part 1

8 May 2021 19:48 UTC
6 points

# So­cial Calibration

20 May 2021 23:22 UTC
3 points

# Be­hav­ior Clon­ing is Miscalibrated

5 Dec 2021 1:36 UTC
77 points

# How the Equiv­a­lent Bet Test Ac­tu­ally Works

18 Dec 2021 11:17 UTC
4 points
(www.erichgrunewald.com)

# [Question] How to best mea­sure if and to what de­gree you’re too pes­simistic or too op­ti­mistic?

31 Mar 2024 0:57 UTC
4 points

# Cal­ibrate—New Chrome Ex­ten­sion for hid­ing num­bers so you can guess

7 Oct 2022 11:21 UTC
59 points

# Fair Col­lec­tive Effi­cient Altruism

25 Nov 2022 9:38 UTC
2 points

# We Change Our Minds Less Often Than We Think

3 Oct 2007 18:14 UTC
100 points

# Break­ing Rank (Cal­ibra­tion Game)

7 Mar 2023 15:40 UTC
11 points

# [Question] Are (Mo­tor)sports like F1 a good thing to cal­ibrate es­ti­mates against?

24 Mar 2024 9:07 UTC
4 points

# Ou­trangeous (Cal­ibra­tion Game)

7 Mar 2023 15:29 UTC
36 points

# Bayes-Up: An App for Shar­ing Bayesian-MCQ

6 Feb 2020 19:01 UTC
53 points

# Plac­ing Your­self as an In­stance of a Class

3 Oct 2017 19:10 UTC
35 points

# Lawful Uncertainty

10 Nov 2008 21:06 UTC
101 points

# Kurzweil’s pre­dic­tions: good ac­cu­racy, poor self-calibration

11 Jul 2012 9:55 UTC
50 points

# Quan­tified In­tu­itions: An epistemics train­ing web­site in­clud­ing a new EA-themed cal­ibra­tion app

20 Sep 2022 22:25 UTC
28 points

# Hor­rible LHC Inconsistency

22 Sep 2008 3:12 UTC
34 points

# Rais­ing the fore­cast­ing wa­ter­line (part 1)

9 Oct 2012 15:49 UTC
51 points