Below is an excerpt of a post I am currently drafting, and I would be curious of whether you have disagreements with something I wrote here. (Mostly point 2 is relevant to you.):
If you want to incentivize truthfulness, you can ask agents for predictions, wait until you can see the result, and score them according to logloss. You can also have them bet in prediction markets. However, there are problems that need to be addressed, most importantly:
Agents betting in prediction markets have incentives to influence the outcome (e.g. by assassinating people) so they can make profit on their insider knowledge of what they will do.
When the prediction is used to make decisions, you run into difficulties. As explained here, in a simple futarchy with conditional prediction markets, the thing that is being bet on is p(outcome | action), whereas what we actually want is p(outcome | do(action)). In particular, agents need to take into account the fact that if an action is chosen, the market probably predicted high p(outcome | action), which leads to confusing self-prediction dynamics. [Footnote: I think (but not sure) in many cases p(outcome | best_predicted_action) may converge to roughly p(outcome | do(best_predicted_action)) very shortly before the action is actually determined. But EDT has some serious flaws like XOR blackmailability that can cause different outcomes and serious problems.] Until shortly before the decision is made (where markets on counterfactual branches get cancelled), the market prices can be very inaccurate[Footnote: This is because action branches, where there’s more information still coming in for predicting how good they would turn out, will be overvalued, since that market-branch is less likely to get cancelled if the information suggests the branch might be favorable. There was an experiment testing one of dynomight’s thought experiments, and it did show this phenomenon. (Note that in the experiment, the final decision of the market was the correct decision (given that it was known that coin B always lands heads). (The decision point is the point where the non-top markets get cancelled.))], and the prices on the counterfactuals may stay inaccurate. [Footnote: It could be that there are cases where (even in theory) futarchy would result in decisions that are insane even from an EDT perspective, although I would very tentatively think this is not the case (but didn’t think much about it).] [Footnote: Here’s one proposed fix to the “futarchy implements EDT” problem, although it has weak-betting-incentives problems. (I’m not sure whether those weak-betting-incentives problems could be addressed well, e.g. by subsidizing the market. Didn’t think about it much.) The proposal works by only predicting cases where you use a fixed randomized intervention protocol for choosing the action, i.e. by having the market not affect decisions in the worlds where it pays out. But there might be other problems where the prediction affects the outcome which cannot be easily sidestepped and a deeper solution here would be desirable. Also, ideally we’d want to implement LDT somehow, not just CDT.]
Very interesting post!
Below is an excerpt of a post I am currently drafting, and I would be curious of whether you have disagreements with something I wrote here. (Mostly point 2 is relevant to you.):