Nah, I don’t think you had mind-killer problems, but as more people wander into the discussion, the potential is there...
For models, it’s not an automatic assumption to assume completeness and mutual exclusion. In particular, mutual exclusion means only one model is right and all the others are wrong—we just don’t know which one. That’s a very black-and-white approach. It’s not as simple but more realistic (for appropriate values of “realistic”) to assume that all models are wrong but some are more wrong than others. Then the correctness of a model is a real number, not a boolean value (and that’s separate from what we know and believe about models), and more, you can then play with dependency structures (e.g. a correlation matrix) which essentially say “if model X is relatively-more-correct then model Y is also relatively-more-correct and model X is relatively-more-wrong”.
Without dependencies this doesn’t look complicated. If the models (and their believability, aka the probability of being the Correct One) are independent then the probabilities are summable. You basically weight the rankings (or scores) by models’ probabilities and you’re good to go. With dependencies it gets more interesting.
I am not sure why you’re questioning what are useful results in this setting. As formulated, the useful result is a list of available choices ranked by estimated utility. Ultimately, you just want to know the top choice, you don’t even care much about how others rank.
Nah, I don’t think you had mind-killer problems, but as more people wander into the discussion, the potential is there...
For models, it’s not an automatic assumption to assume completeness and mutual exclusion. In particular, mutual exclusion means only one model is right and all the others are wrong—we just don’t know which one. That’s a very black-and-white approach. It’s not as simple but more realistic (for appropriate values of “realistic”) to assume that all models are wrong but some are more wrong than others. Then the correctness of a model is a real number, not a boolean value (and that’s separate from what we know and believe about models), and more, you can then play with dependency structures (e.g. a correlation matrix) which essentially say “if model X is relatively-more-correct then model Y is also relatively-more-correct and model X is relatively-more-wrong”.
Without dependencies this doesn’t look complicated. If the models (and their believability, aka the probability of being the Correct One) are independent then the probabilities are summable. You basically weight the rankings (or scores) by models’ probabilities and you’re good to go. With dependencies it gets more interesting.
I am not sure why you’re questioning what are useful results in this setting. As formulated, the useful result is a list of available choices ranked by estimated utility. Ultimately, you just want to know the top choice, you don’t even care much about how others rank.