“Community Notes” resolution for vague predictions
Is there some kind of “get prediction markets, or predictions, onto twitter as a central object” project going? If so, how is it going?
I’m thinking through “how to raise median sanity” on a world scale. There are several incentive and institutional problems that make this very difficult. One angle is “try to make it a thing that the world tracks and cares about your predictions, and getting them right/wrong.”
Two past angles here were:
Fact checker sites of the 90s/00s, which became politicized
Prediction markets, which I feel like aren’t a good fit for most of what people would/should care about here. Operationalizing them is extremely annoying.
An idea for a twitter feature, which I think Musk should conceptually like given his stated values, and, I think he might have enough inertia to get behind, is:
When you make a prediction shaped tweet, it automatically gets converted into a prediction-y object*. AI suggests a few plausible operationalizations, or specific edge cases that seem likely to come up that you might want to address.
People can “like” predictions, predictions with a lot of attention get more automated followup.
Instead of resolving predictions with explicit careful operationalization, people vote on whether it seems like it resolved, and a community-notes-esque algorithm handles partisan disagreements.
popular predictions automatically get AI-generated community-notes-eque followup in terms of collecting evidence.
So, the final result will give a succinct “do The People think this resolved true?”, which is some kind of percentage score that takes into account “how much it seems true to people” (if it’s ambiguous / on-a-spectrum), and controversial the answer is. (TODO: figure out good mathy operationalization that is robust to partisanship)
But then, immediately underneath that, is a list of citations/evidence, so even if something is ambiguous, it’s as easy as possible to see concrete facts related to it. (Also with community-notes-esque voting on relevance)
“Have you successfully made good predictions” is a badge / leaderboard that is optimized to be fun and rewarding.
The “community-notes-for-vague-prediction-resolution” mechanic could be implemented by Manifold or Metaculus or LessWrong or whoever, but, I think it’s a lot more useful if it’s on twitter. (tho maybe good to test it out on these places to iron kinks out and prototype UI).
My impression of the landscape is “Musk did really care about but truth, and believes in community notes in particular, but also went crazy after buying twitter. The Community Notes team is a small fiefdom he trusts and still believes in.” So, insofar as this is pitched as an extension of that, might be doable.
Curious about takes from @Zvi and @Austin Chen.
Low hanging fruit: There used to be a bot (“RemindMeOfThis” or similar) people could tag to get a reminder in some time frame. For some reason I don’t see a currently working version of this. I assume a contributing factor is that the X API is now expensive.
To be clear, your proposal does not include a prediction market of any sort, just AI automated operationalizations + user voted results?
I think giving twitter users some internet points they can use to bet on the markets, like with manifold, may just work. But it’s possible that the public won’t use that ir like that compared to user voting.
Yeah my impression is that prediction markets incentivize things that are more like “seek out poorly priced things and arbitrage them” rather than “make useful, correct predictions that anyone cared about.” (I have some vague sense that the shape of prediction markets tends towards not solving the problems I care about, although I haven’t thought about it too hard).
There totally could also be prediction markets involved here. But the primary purpose is to establish a record of “who predicted accurate things that anyone cared about” so that can become an easily referenced source-of-authority.
This needs to be incentive-compatible with “being a pundit”, I think. Like the idea is to make it so people who do pundrity on twitter are automatically nudged to make their predictions legible and gradable.
The default is of course that people who do punditry are automatically nudged to make their predictions illegible, because the existence of any such system threatens to embarrass the many pundits who know their predictions are bullshit.
This seems like a huge problem and I don’t know how you get around it, other than hoping the system takes off so well that people can be shamed for conspicuously dodging its accountability.
I agree it’s quite difficult to solve this completely. But, it seems like you could make it so “On twitter, the default thing is if you try to pundit, the system nudges you towards legiblizing your predictions.”
This at least makes more of the world, on the margin, track “being correct” as a good thing to do. This can hopefully have flowthrough effects as the equilibrium shifts from “illegible pundits just look like status quo” to “illegible pundits look weirder.”