Is there a way to incentivize reporting true probability distribution? Say I Bob wants Alice to provide her probability distribution of IQ she’ll get on the test. He is willing to give her a real number as a reward, he wants to hear her probability distribution of her result. What should he do?
Would be nice if it worked for both discrete and non-discrete probability spaces.
In the discrete case log scoring still works, it generalizes past the binary case.
That is, if S is the set of possible outcomes of the test, Bob elicits from Alice a probability distribution q(s) on S, then Alice takes the test and gets some outcome s∈S, then Bob rewards Alice logq(s). (This number is unfortunately always negative; you can add a positive constant to it if you want.)
Alice’s expected payoff according to her true probability distribution p(s) is
∑s∈Sp(s)logq(s)
also known as the (negative of the) cross entropy between p and q. And you can do a computation, e.g. with Lagrange multipliers, which will verify that for fixed p, the optimal value of q is q=p. I do this calculation in this blog post.
A test isn’t a good example to use because the outcome of the test is under Alice’s control, so she can e.g. throw the test and predict this fact. This procedure is best used to elicit Alice’s prediction of something which she cannot influence in any way.
I tried starting with a dollar sign, which brought up a yellow prompt that I couldn’t figure out how to easily exit; hitting Enter just started a new line in the prompt. The only way I’ve found to exit it so far is Ctrl + Enter, which submits the comment with the LaTeX displaying as “refresh to display LaTeX,” and continuing to display that after I refresh.
Ah, you exit the yellow prompt with Esc and in the yellow prompt you can type any LaTeX, with a live-preview beneath it.
Somewhat surprised that it continued to show “refresh to display LaTeX’ even after you refreshed. I never had that happen to me. That might have been a result of you submitting from the inside of the prompt, which I can imagine causing errors.
Is there a way to incentivize reporting true probability distribution? Say I Bob wants Alice to provide her probability distribution of IQ she’ll get on the test. He is willing to give her a real number as a reward, he wants to hear her probability distribution of her result. What should he do?
Would be nice if it worked for both discrete and non-discrete probability spaces.
In the discrete case log scoring still works, it generalizes past the binary case.
That is, if S is the set of possible outcomes of the test, Bob elicits from Alice a probability distribution q(s) on S, then Alice takes the test and gets some outcome s∈S, then Bob rewards Alice logq(s). (This number is unfortunately always negative; you can add a positive constant to it if you want.)
Alice’s expected payoff according to her true probability distribution p(s) is
∑s∈Sp(s)logq(s)
also known as the (negative of the) cross entropy between p and q. And you can do a computation, e.g. with Lagrange multipliers, which will verify that for fixed p, the optimal value of q is q=p. I do this calculation in this blog post.
A test isn’t a good example to use because the outcome of the test is under Alice’s control, so she can e.g. throw the test and predict this fact. This procedure is best used to elicit Alice’s prediction of something which she cannot influence in any way.
How did using LaTeX fail?
I tried starting with a dollar sign, which brought up a yellow prompt that I couldn’t figure out how to easily exit; hitting Enter just started a new line in the prompt. The only way I’ve found to exit it so far is Ctrl + Enter, which submits the comment with the LaTeX displaying as “refresh to display LaTeX,” and continuing to display that after I refresh.
Ah, you exit the yellow prompt with
Escand in the yellow prompt you can type any LaTeX, with a live-preview beneath it.Somewhat surprised that it continued to show “refresh to display LaTeX’ even after you refreshed. I never had that happen to me. That might have been a result of you submitting from the inside of the prompt, which I can imagine causing errors.
Awesome, everything’s fine now.