The space of possible DRT-induction method pairs is much larger than this would suggest.
I think the space of things you could try is quite large indeed, both when it comes to DRT-induction as well as what you choose to include in the control condition. I can also imagine this being a major point of contention/annoyance post-study (“This is nice, but for me to really change my mind I’d want you to have used this induction/control”).
Solution
Before the experiment, we have prediction markets/forecasting tournaments on the results of the pre-registered statistical tests, given a particular induction x control combination.
When the markets close, your experiment runs as planned—but you only test the induction x control combinations that had the most disagreement/variance in their estimates.
Prediction market participants are then paid according to a proper scoring rule based on the outcome of the experiment.
So overall, even if you just test 1-3 experimental designs, we could have these markets on 10-20 designs, and get priors for all of them!
This is also a more transparent way of picking conditions to run for the experiment.
___
I’ve messaged you privately to discuss this further and organise eventual funding and operational support.
Problem
I think the space of things you could try is quite large indeed, both when it comes to DRT-induction as well as what you choose to include in the control condition. I can also imagine this being a major point of contention/annoyance post-study (“This is nice, but for me to really change my mind I’d want you to have used this induction/control”).
Solution
Before the experiment, we have prediction markets/forecasting tournaments on the results of the pre-registered statistical tests, given a particular induction x control combination.
When the markets close, your experiment runs as planned—but you only test the induction x control combinations that had the most disagreement/variance in their estimates.
Prediction market participants are then paid according to a proper scoring rule based on the outcome of the experiment.
So overall, even if you just test 1-3 experimental designs, we could have these markets on 10-20 designs, and get priors for all of them!
This is also a more transparent way of picking conditions to run for the experiment.
___
I’ve messaged you privately to discuss this further and organise eventual funding and operational support.