So, what do you think? Does this method seem at all promising? I’m debating with myself whether I should begin using SPIES on Metaculus or elsewhere.
I’m not super impressed tbh. I don’t see “give a 90% confidence interval for x” as a question which comes up frequently? (At least in the context of eliciting forecasts and estimates from humans—it comes up quite a bit in data analysis).
For example, I don’t really understand how you’d use it as a method on Metaculus. Metaculus has 2 question types—binary and continuous. For binary you have to give the probability an event happens—not sure how you’d use SPIES to help here. For continuous you are effectively doing the first step of SPIES—specifying the full distribution.
If I was to make a positive case for this, it would be—forcing people to give a full distribution results in better forecasts for sub-intervals. This seems an interesting (and plausible claim) but I don’t find anything beyond that insight especially valuable.
I’m not super impressed tbh. I don’t see “give a 90% confidence interval for x” as a question which comes up frequently? (At least in the context of eliciting forecasts and estimates from humans—it comes up quite a bit in data analysis).
For example, I don’t really understand how you’d use it as a method on Metaculus. Metaculus has 2 question types—binary and continuous. For binary you have to give the probability an event happens—not sure how you’d use SPIES to help here. For continuous you are effectively doing the first step of SPIES—specifying the full distribution.
If I was to make a positive case for this, it would be—forcing people to give a full distribution results in better forecasts for sub-intervals. This seems an interesting (and plausible claim) but I don’t find anything beyond that insight especially valuable.