This seems pretty reasonable. In particular I think you’ve very clearly articulated a set of principles which are valuable for prediction. However, I feel like your principles for forecasting seem to explicitly lead to most forecasting needing to be rules based. For example, to begin with you say:
In short, when making rules, one can front-load intentions by writing details upfront, or back-load work by stating high-level principles and having procedures to decide on details on an as-needed basis*. [...] I think that the question shouldn’t be implicitly decided by front loading assumptions, which is often the current default. More than that, I think the balance should be better and more explicitly be addressed.
But then later on you say:
This requires that they be resolvable.
The resolution criteria should be well-specified.
If relevant or possible, the intent of the question, or guidance for how resolution will occur, should be clear.
Ambiguity should be avoided, but by default, when (inevitable) ambiguity arises, intent should guide the resolution. Any guidance about the motive or intent of the question can therefore be an asset. This is especially true when resolutions are based on expert opinion.
By default, forecasts should be assumed to be about object-level issues.
My first take reading “decide on details on an as-needed basis” was that we could somehow avoid most of the pain which goes alongside writing forecasting questions, but that doesn’t seem to be the case.
Another example would be the need to have explicit rules for scoring.
To give a concrete example, I think Metaculus is pretty far to the “Rules” end of the spectrum in your frame. Despite this, I can’t really see what you’re suggesting would change if they shifted to a more “Principles” based approach. If I had to speculate, I would guess your only “real” change would be something like:
“Where the spirit and letter of a question conflict, the question should be resolved based on the spirit”
This seems pretty reasonable to me (although I don’t feel that you’ve made a compelling case for it).
----
But for prediction markets to be useful, there should be a balance between principles and rules.
So I don’t fully agree with the logic in this section. I agree that often the spirit and letter of a question can disagree, but I don’t necessarily think that prediction markets will be “more useful” if they have a more principles based approach. (At least, I think the downsides from potentially uncertain resolution outweigh the upside from potentially resolving “more correctly”). [To give a concrete example, my comfort level using Polymarket decreased substantially when they (in my opinion) changed their resolution criteria on their US Election market. (When the on-ramp and off-ramp started to creak that was the nail in the coffin as far as I was concerned)]
I haven’t said, and I don’t think, that the majority of markets and prediction sites get this wrong. I think they navigate this without a clear framework, which I think the post begins providing. And I strongly agree that there isn’t a slam-dunk-no-questions case for principles overriding rules, which the intro might have implied too strongly. I also agree with your point about downsides of ambiguity potentially overriding the benefits of greater fidelity to the intent of a question, and brought it up in the post. Still, excessive focus on making rules on the front end, especially for longer-term questions and ones where the contours are unclear, rather than explicitly being adaptive, is not universally helpful.
And clarifications that need to change the resolution criteria mid-way are due to either bad questions, or badly handled resolutions. At the same time, while there are times that avoiding ambiguity is beneficial, there are also times when explicitly addressing corner cases to make them unambiguous (“if the data is discontinued or the method is changed, the final value posted using the current method will be used”) makes the question worse, rather than better.
Lastly, I agree that one general point I didn’t say, but agree with, was that “where the spirit and letter of a question conflict, the question should be resolved based on the spirit.” I mostly didn’t make an explicit case for this because I think it’s under-specified as a claim. Instead, the three more specific claims I would make are: 1) When the wording of a question seems ambiguous, the intent should be an overriding reason to choose an interpretation. 2) When the wording of a question is clear, the intent shouldn’t change the resolution.
(I realise everything I’m commenting seems like a nitpick and I do think that what you’ve written is interesting and useful, I just don’t have anything constructive to add on that side of things)
I don’t like litigating via quotes, but:
I haven’t said, and I don’t think, that the majority of markets and prediction sites get this wrong.
and
More than that, I think the balance should be better and more explicitly be addressed.
I read the bit I’ve emphasised as saying “prediction sites have got this balance wrong” contradicting your comment saying you think they have it right.
Still, excessive focus on making rules on the front end, especially for longer-term questions and ones where the contours are unclear, rather than explicitly being adaptive, is not universally helpful.
I think it’s really hard for this adaptive approach to work when there’s more than a small group of like minded people involved in a forecast. (This is related to my final point):
1) When the wording of a question seems ambiguous, the intent should be an overriding reason to choose an interpretation. 2) When the wording of a question is clear, the intent shouldn’t change the resolution.
The problem for me (with this) is what is “clear” for some people is not clear for others. To give one example of this, the language in this question was completely unambiguous to me (and it’s author) but another predictor found it unclear. (I don’t think this is a particularly good example, but it’s just one which I thought of when trying to think of an example of where some people thought something was ambiguous and some people didn’t).
re: “Get this wrong” versus “the balance should be better,” there are two different things that are being discussed. The first is about defining individual questions via clear resolution criteria, which I think is doe well, and the second is about defining clear principles that provide context and inform what types of questions and resolution criteria are considered good form.
A question like “will Democrats pass H.R.2280 and receive 51 votes in the Senate” is very well defined, but super-narrow, and easily resolved “incorrectly” if the bill is incorporated into another bill, or if an adapted bill is proposed by a moderate Republican and passes instead, or passed via some other method, or if it passes but gets vetoed by Biden. But it isn’t an unclear question, and given the current way that Metaculus is run, would probably be the best way of phrasing the question. Still, it’s a sub-par question, given the principles I mentioned. A better one would be “Will a bill such as H.R.2280 limiting or banning straw purchases of firearms be passed by the current Congress and enacted?” It’s much less well defined, but the boundaries are very different. It also uses “passed” and “enacted”, which have gray areas. At the same time, the failure modes are closer to the ones that we care about near the boundary of the question. However, given the current system, this question is obviously worse—it’s harder to resolve, it’s more likely to be ambiguous because a bill that does only some of the thing we care about is passed, etc.
Still, I agree that the boundaries here are tricky, and I’d love to think more about how to do this better.
This seems pretty reasonable. In particular I think you’ve very clearly articulated a set of principles which are valuable for prediction. However, I feel like your principles for forecasting seem to explicitly lead to most forecasting needing to be rules based. For example, to begin with you say:
But then later on you say:
My first take reading “decide on details on an as-needed basis” was that we could somehow avoid most of the pain which goes alongside writing forecasting questions, but that doesn’t seem to be the case.
Another example would be the need to have explicit rules for scoring.
To give a concrete example, I think Metaculus is pretty far to the “Rules” end of the spectrum in your frame. Despite this, I can’t really see what you’re suggesting would change if they shifted to a more “Principles” based approach. If I had to speculate, I would guess your only “real” change would be something like:
“Where the spirit and letter of a question conflict, the question should be resolved based on the spirit”
This seems pretty reasonable to me (although I don’t feel that you’ve made a compelling case for it).
----
So I don’t fully agree with the logic in this section. I agree that often the spirit and letter of a question can disagree, but I don’t necessarily think that prediction markets will be “more useful” if they have a more principles based approach. (At least, I think the downsides from potentially uncertain resolution outweigh the upside from potentially resolving “more correctly”). [To give a concrete example, my comfort level using Polymarket decreased substantially when they (in my opinion) changed their resolution criteria on their US Election market. (When the on-ramp and off-ramp started to creak that was the nail in the coffin as far as I was concerned)]
I haven’t said, and I don’t think, that the majority of markets and prediction sites get this wrong. I think they navigate this without a clear framework, which I think the post begins providing. And I strongly agree that there isn’t a slam-dunk-no-questions case for principles overriding rules, which the intro might have implied too strongly. I also agree with your point about downsides of ambiguity potentially overriding the benefits of greater fidelity to the intent of a question, and brought it up in the post. Still, excessive focus on making rules on the front end, especially for longer-term questions and ones where the contours are unclear, rather than explicitly being adaptive, is not universally helpful.
And clarifications that need to change the resolution criteria mid-way are due to either bad questions, or badly handled resolutions. At the same time, while there are times that avoiding ambiguity is beneficial, there are also times when explicitly addressing corner cases to make them unambiguous (“if the data is discontinued or the method is changed, the final value posted using the current method will be used”) makes the question worse, rather than better.
Lastly, I agree that one general point I didn’t say, but agree with, was that “where the spirit and letter of a question conflict, the question should be resolved based on the spirit.” I mostly didn’t make an explicit case for this because I think it’s under-specified as a claim. Instead, the three more specific claims I would make are:
1) When the wording of a question seems ambiguous, the intent should be an overriding reason to choose an interpretation.
2) When the wording of a question is clear, the intent shouldn’t change the resolution.
(I realise everything I’m commenting seems like a nitpick and I do think that what you’ve written is interesting and useful, I just don’t have anything constructive to add on that side of things)
I don’t like litigating via quotes, but:
and
I read the bit I’ve emphasised as saying “prediction sites have got this balance wrong” contradicting your comment saying you think they have it right.
I think it’s really hard for this adaptive approach to work when there’s more than a small group of like minded people involved in a forecast. (This is related to my final point):
The problem for me (with this) is what is “clear” for some people is not clear for others. To give one example of this, the language in this question was completely unambiguous to me (and it’s author) but another predictor found it unclear. (I don’t think this is a particularly good example, but it’s just one which I thought of when trying to think of an example of where some people thought something was ambiguous and some people didn’t).
re: “Get this wrong” versus “the balance should be better,” there are two different things that are being discussed. The first is about defining individual questions via clear resolution criteria, which I think is doe well, and the second is about defining clear principles that provide context and inform what types of questions and resolution criteria are considered good form.
A question like “will Democrats pass H.R.2280 and receive 51 votes in the Senate” is very well defined, but super-narrow, and easily resolved “incorrectly” if the bill is incorporated into another bill, or if an adapted bill is proposed by a moderate Republican and passes instead, or passed via some other method, or if it passes but gets vetoed by Biden. But it isn’t an unclear question, and given the current way that Metaculus is run, would probably be the best way of phrasing the question. Still, it’s a sub-par question, given the principles I mentioned. A better one would be “Will a bill such as H.R.2280 limiting or banning straw purchases of firearms be passed by the current Congress and enacted?” It’s much less well defined, but the boundaries are very different. It also uses “passed” and “enacted”, which have gray areas. At the same time, the failure modes are closer to the ones that we care about near the boundary of the question. However, given the current system, this question is obviously worse—it’s harder to resolve, it’s more likely to be ambiguous because a bill that does only some of the thing we care about is passed, etc.
Still, I agree that the boundaries here are tricky, and I’d love to think more about how to do this better.