Epistemic status: I am probably misunderstanding some critical parts of the theory, and I am quite ignorant on technical implementation of prediction markets. But posting this could be useful for my and others’ learning.
First question. Am I understanding correctly how the market would function. Taking your IRT probit market example, here is what I gather:
(1) I want to make a bet on the conditional market P(X_i | Y). I have a visual UI where I slide bars to make a bet on parameters a and b; (dropping subscript i) however, internally this is represented by a bet on a’ = a sigma_y and P(X) = Phi(b’/sqrt(a’+1)), b’ = b + a mu_y. So far, so clear.
(2) I want to bet on the latent market P(Y). I make bet on mu_Y and sigma_y, which is internally represented by a bet on a’ and P(X). In your demo, this is explicit: P(Y) actually remains a standard normal distribution, it is the conditional distributions that shift.
(3) I want to bet on the unconditional market P(X_i), which happens to be an indicator variable for some latent variable market I don’t really care about. I want to make a bet on P(X_i) only. What does exactly happen? If the internal technical representation is a database row [p_i, a’] changing to [p_new, a’], then I must be implicitly making a bet on a’ and b’ too, as b’ derived from a and P(X) and P(X) changed. In other words, it appears I am also making a bet on the conditional market?
Maybe this is OK if I simply ignore the latent variable market, but I could also disagree with it: I think P(X) is not correlated with the other indicators, or the correlation structure implied by linear probit IRT model is inadequate. Can I make a bet of form [p_i, NA] and have it not to participate in the latent variable market structure? (Ie when the indicator market resolves, p_i is taken into account for scoring, NA is ignored while scoring Y).
Thus, I can launch a competing latent variable market for the same indicators? How can I make a bet against a particular latent structure in favor of another structure? A model selection bet, if you will.
Context for the question: In my limited experience of latent variable models for statistical inference tasks, the most difficult and often contested question is whether the latent structure you specified is good one. The choice of model and interpretations of it are very likely to get criticized and debated in published work.
Another question for discussion.
It seems theoretically possible to obtain individual predictors’ bets / predictions on X_1 … X_k, without presence of any latent variable market, impute for the missing values if some predictors have not predicted on all k indicators, and then estimate a latent variable model on this data. What is the exact benefit of having the latent model available for betting? If fit a Bayesian model (non-market LVM with indicator datapoints) and it converges, from this non-market model, I would obtain posterior distribution for parameters and compute many of the statistics of interest.
Presumably, if there is a latent “ground truth” relationship between all indicator questions and the market participants are considering them, the relationship would be recovered by such analysis. If the model is non-identifiable or misspecified (= such ground truth does not exist / is not recoverable), I will have problems fitting it and with the interpretation of fit. (And if I were to run a prediction market for a badly specified LVM, the market would exhibit problematic behavior.)
By not making a prediction market for the latent model available, I won’t have predictions on the latent distribution directly, but on the other hand, I could try to estimate several different without imposing any of their assumptions about presumed joint structure on the market. I could see this beneficial, depending on how the predictions on Y “propagate” to indicator markets or the other way around (case (3) above).
Final question for discussion: Suppose I run a prediction market using the current market implementations (Metaculus or Manifold or something else, without any LVM support), where I promise to fit a particular IRT probit model at time point t in future and ask for predictions on the distribution of posterior model parameters estimates when I fit the model. Wouldn’t I obtain most of the benefits of “native technical implementation” of a LVM market … except that updating away any inconsistencies between the LVM prediction market estimate and individual indicator market will be up to to individual market participants? The bets on this estimated LVM market at least should behave as any other bets, right? (Related: which should be the time point t when I promise to fit the model and resolve the market? t = when the first indicator market resolves? when the last indicator market resolves? arbitrary fixed time point T?)
(1) I want to make a bet on the conditional market P(X_i | Y). I have a visual UI where I slide bars to make a bet on parameters a and b; (dropping subscript i) however, internally this is represented by a bet on a’ = a sigma_y and P(X) = Phi(b’/sqrt(a’+1)), b’ = b + a mu_y. So far, so clear.
Yep.
(Or maybe not a and b, but some other parameterization; I would guess the most intuitive UI is still an open question.)
(2) I want to bet on the latent market P(Y). I make bet on mu_Y and sigma_y, which is internally represented by a bet on a’ and P(X). In your demo, this is explicit: P(Y) actually remains a standard normal distribution, it is the conditional distributions that shift.
Yep.
(3) I want to bet on the unconditional market P(X_i), which happens to be an indicator variable for some latent variable market I don’t really care about. I want to make a bet on P(X_i) only. What does exactly happen? If the internal technical representation is a database row [p_i, a’] changing to [p_new, a’], then I must be implicitly making a bet on a’ and b’ too, as b’ derived from a and P(X) and P(X) changed. In other words, it appears I am also making a bet on the conditional market?
Maybe this is OK if I simply ignore the latent variable market, but I could also disagree with it: I think P(X) is not correlated with the other indicators, or the correlation structure implied by linear probit IRT model is inadequate. Can I make a bet of form [p_i, NA] and have it not to participate in the latent variable market structure? (Ie when the indicator market resolves, p_i is taken into account for scoring, NA is ignored while scoring Y).
Thus, I can launch a competing latent variable market for the same indicators? How can I make a bet against a particular latent structure in favor of another structure? A model selection bet, if you will.
Context for the question: In my limited experience of latent variable models for statistical inference tasks, the most difficult and often contested question is whether the latent structure you specified is good one. The choice of model and interpretations of it are very likely to get criticized and debated in published work.
Good question! This is actually a problem that I didn’t realize at the time I started promoting LVPMs, though I did realize it somewhere along the way, albeit underestimating it and expecting it to solve itself, though somewhere along the way I realized that I had underestimated it. I still think it is only a very minor problem, but lemme give my analysis:
There are multiple ways to implement this type of bet, and your approach of changing [p_old, a’] to [p_new, a’] is only one of them; it was the one I favored earlier on, but there’s an alternate approach that others have favored, and in fact there’s a first approach that I’ve now come to favor. Let’s take all of them.
The approach described by my post/by you: Assuming that we change the internal representation in the database from [p_old, a’] to [p_new, a’], this is equivalent to making a bet on the latent variable market that only affects b. So essentially you would be making a bet that b has higher probability across the entire latent variable.
As part of my initial push, I made the mistake of assuming that this was equivalent to just making a bet on X without the latent variable market, but then later I realized I was mixing up negative log probs and logits (you are making a bet that b has higher logit across the latent variable, but a prediction market without latent variables would be making a bet that b has higher negative log probabilities, not higher logits).
Conventional approach: I think the approach advocated by e.g. Robin Hanson is to have one grand Bayes net structure for all of the markets that anyone can make predictions on, and then when people make a prediction on only a subset of the structure, such as an indicator variable, then that bet is treated as virtual evidence which is used to update the entire Bayesian network.
This is basically similar in consequences to what you describe: if people make a bet on some indicator variable, they make implied bets on lots of other things too. However the implied bets are more complicated and less risky to the better than the version described in my post. (When I had written the post, I hadn’t realized those complexities.)
In order to make a model selection bet, you would bet directly on the Bayes net structure that hangs around all of the markets, rather than betting on X directly.
I think the conventional approach has traditionally mainly struggled because it’s had to deal with making Bayesian updates on a huge Bayes net. I would prefer something more atomic/discrete, and I would not be surprised if Manifold Markets would prefer something more atomic/discrete too. Treating it as Bayesian updates is also a bit of a problem for certain latent variable models, since e.g. if you fix the distribution of your latent variable to be Gaussian, then there are certain Bayesian updates that you cannot exactly do.
My new preferred approach: Rather than thinking of the LVPM as sharing parameters with the indicator markets (i.e. the (P(X),a′) model), I’d probably be inclined to give it separate parameters (i.e. a (P(X),a′,b′) model). Of course then the trouble is that we could enter into an inconsistent state where the probabilities implied by the LVPM do not equal the probabilities from the indicator markets.
Ideally to minimize confusion and maximize information flow, we would like to go from an inconsistent state to a consistent state as quickly as possible. I think this could be done with an arbitrage bot (it seems like most of the time this would probably be +EV, so the bots should remain solvent), though perhaps the market should offer users to automatically make the arbitrage bets themselves so they can benefit from the presence of the latent variables.
I expect to give an update on this later with more formal derivations, likely together with my implementation of latent class markets for Manifold.
It seems theoretically possible to obtain individual predictors’ bets / predictions on X_1 … X_k, without presence of any latent variable market, impute for the missing values if some predictors have not predicted on all k indicators, and then estimate a latent variable model on this data. What is the exact benefit of having the latent model available for betting? If fit a Bayesian model (non-market LVM with indicator datapoints) and it converges, from this non-market model, I would obtain posterior distribution for parameters and compute many of the statistics of interest.
Presumably, if there is a latent “ground truth” relationship between all indicator questions and the market participants are considering them, the relationship would be recovered by such analysis. If the model is non-identifiable or misspecified (= such ground truth does not exist / is not recoverable), I will have problems fitting it and with the interpretation of fit. (And if I were to run a prediction market for a badly specified LVM, the market would exhibit problematic behavior.)
By not making a prediction market for the latent model available, I won’t have predictions on the latent distribution directly, but on the other hand, I could try to estimate several different without imposing any of their assumptions about presumed joint structure on the market. I could see this beneficial, depending on how the predictions on Y “propagate” to indicator markets or the other way around (case (3) above).
Strictly speaking, it’s an empirical question how well this would work.
I could certainly imagine programmatically fitting a latent variable model to prediction market data, assuming that there is a fixed set of outcomes and the users mainly disagree about the latent variable values. However, it seems to me that this would run into massive problems due to e.g. Kelly betting where people don’t update the probabilities all the way to match their beliefs. You could probably attempt to model the participant’s risk aversion etc., but I have a hard time believing that it would work.
Final question for discussion: Suppose I run a prediction market using the current market implementations (Metaculus or Manifold or something else, without any LVM support), where I promise to fit a particular IRT probit model at time point t in future and ask for predictions on the distribution of posterior model parameters estimates when I fit the model. Wouldn’t I obtain most of the benefits of “native technical implementation” of a LVM market … except that updating away any inconsistencies between the LVM prediction market estimate and individual indicator market will be up to to individual market participants? The bets on this estimated LVM market at least should behave as any other bets, right?
Asking for predictions about the parameters of the model would make it more like a method of moments estimator than a maximum likelihood estimator, right? Or maybe not, it seems kind of hard to predict how exactly it would go.
Regardless, I don’t think this is an unreasonable idea to experiment with, but I think there are two challenges:
As mentioned, I’m pretty skeptical of the notion that you can fit LV models to prediction market predictions and have it work well.
If you don’t have LVPM support, the UI for making predictions would probably be even worse than the UI I made.
I think if this was somehow changed to be maximum likelihood, and if those two challenges were removed, then you would get something close to “My new preferred approach”.
Epistemic status: I am probably misunderstanding some critical parts of the theory, and I am quite ignorant on technical implementation of prediction markets. But posting this could be useful for my and others’ learning.
First question. Am I understanding correctly how the market would function. Taking your IRT probit market example, here is what I gather:
(1) I want to make a bet on the conditional market P(X_i | Y). I have a visual UI where I slide bars to make a bet on parameters a and b; (dropping subscript i) however, internally this is represented by a bet on a’ = a sigma_y and P(X) = Phi(b’/sqrt(a’+1)), b’ = b + a mu_y. So far, so clear.
(2) I want to bet on the latent market P(Y). I make bet on mu_Y and sigma_y, which is internally represented by a bet on a’ and P(X). In your demo, this is explicit: P(Y) actually remains a standard normal distribution, it is the conditional distributions that shift.
(3) I want to bet on the unconditional market P(X_i), which happens to be an indicator variable for some latent variable market I don’t really care about. I want to make a bet on P(X_i) only. What does exactly happen? If the internal technical representation is a database row [p_i, a’] changing to [p_new, a’], then I must be implicitly making a bet on a’ and b’ too, as b’ derived from a and P(X) and P(X) changed. In other words, it appears I am also making a bet on the conditional market?
Maybe this is OK if I simply ignore the latent variable market, but I could also disagree with it: I think P(X) is not correlated with the other indicators, or the correlation structure implied by linear probit IRT model is inadequate. Can I make a bet of form [p_i, NA] and have it not to participate in the latent variable market structure? (Ie when the indicator market resolves, p_i is taken into account for scoring, NA is ignored while scoring Y).
Thus, I can launch a competing latent variable market for the same indicators? How can I make a bet against a particular latent structure in favor of another structure? A model selection bet, if you will.
Context for the question: In my limited experience of latent variable models for statistical inference tasks, the most difficult and often contested question is whether the latent structure you specified is good one. The choice of model and interpretations of it are very likely to get criticized and debated in published work.
Another question for discussion.
It seems theoretically possible to obtain individual predictors’ bets / predictions on X_1 … X_k, without presence of any latent variable market, impute for the missing values if some predictors have not predicted on all k indicators, and then estimate a latent variable model on this data. What is the exact benefit of having the latent model available for betting? If fit a Bayesian model (non-market LVM with indicator datapoints) and it converges, from this non-market model, I would obtain posterior distribution for parameters and compute many of the statistics of interest.
Presumably, if there is a latent “ground truth” relationship between all indicator questions and the market participants are considering them, the relationship would be recovered by such analysis. If the model is non-identifiable or misspecified (= such ground truth does not exist / is not recoverable), I will have problems fitting it and with the interpretation of fit. (And if I were to run a prediction market for a badly specified LVM, the market would exhibit problematic behavior.)
By not making a prediction market for the latent model available, I won’t have predictions on the latent distribution directly, but on the other hand, I could try to estimate several different without imposing any of their assumptions about presumed joint structure on the market. I could see this beneficial, depending on how the predictions on Y “propagate” to indicator markets or the other way around (case (3) above).
Final question for discussion: Suppose I run a prediction market using the current market implementations (Metaculus or Manifold or something else, without any LVM support), where I promise to fit a particular IRT probit model at time point t in future and ask for predictions on the distribution of posterior model parameters estimates when I fit the model. Wouldn’t I obtain most of the benefits of “native technical implementation” of a LVM market … except that updating away any inconsistencies between the LVM prediction market estimate and individual indicator market will be up to to individual market participants? The bets on this estimated LVM market at least should behave as any other bets, right? (Related: which should be the time point t when I promise to fit the model and resolve the market? t = when the first indicator market resolves? when the last indicator market resolves? arbitrary fixed time point T?)
Yep.
(Or maybe not a and b, but some other parameterization; I would guess the most intuitive UI is still an open question.)
Yep.
Good question! This is actually a problem that I didn’t realize at the time I started promoting LVPMs, though I did realize it somewhere along the way, albeit underestimating it and expecting it to solve itself, though somewhere along the way I realized that I had underestimated it. I still think it is only a very minor problem, but lemme give my analysis:
There are multiple ways to implement this type of bet, and your approach of changing [p_old, a’] to [p_new, a’] is only one of them; it was the one I favored earlier on, but there’s an alternate approach that others have favored, and in fact there’s a first approach that I’ve now come to favor. Let’s take all of them.
The approach described by my post/by you: Assuming that we change the internal representation in the database from [p_old, a’] to [p_new, a’], this is equivalent to making a bet on the latent variable market that only affects b. So essentially you would be making a bet that b has higher probability across the entire latent variable.
As part of my initial push, I made the mistake of assuming that this was equivalent to just making a bet on X without the latent variable market, but then later I realized I was mixing up negative log probs and logits (you are making a bet that b has higher logit across the latent variable, but a prediction market without latent variables would be making a bet that b has higher negative log probabilities, not higher logits).
Conventional approach: I think the approach advocated by e.g. Robin Hanson is to have one grand Bayes net structure for all of the markets that anyone can make predictions on, and then when people make a prediction on only a subset of the structure, such as an indicator variable, then that bet is treated as virtual evidence which is used to update the entire Bayesian network.
This is basically similar in consequences to what you describe: if people make a bet on some indicator variable, they make implied bets on lots of other things too. However the implied bets are more complicated and less risky to the better than the version described in my post. (When I had written the post, I hadn’t realized those complexities.)
In order to make a model selection bet, you would bet directly on the Bayes net structure that hangs around all of the markets, rather than betting on X directly.
I think the conventional approach has traditionally mainly struggled because it’s had to deal with making Bayesian updates on a huge Bayes net. I would prefer something more atomic/discrete, and I would not be surprised if Manifold Markets would prefer something more atomic/discrete too. Treating it as Bayesian updates is also a bit of a problem for certain latent variable models, since e.g. if you fix the distribution of your latent variable to be Gaussian, then there are certain Bayesian updates that you cannot exactly do.
My new preferred approach: Rather than thinking of the LVPM as sharing parameters with the indicator markets (i.e. the (P(X),a′) model), I’d probably be inclined to give it separate parameters (i.e. a (P(X),a′,b′) model). Of course then the trouble is that we could enter into an inconsistent state where the probabilities implied by the LVPM do not equal the probabilities from the indicator markets.
Ideally to minimize confusion and maximize information flow, we would like to go from an inconsistent state to a consistent state as quickly as possible. I think this could be done with an arbitrage bot (it seems like most of the time this would probably be +EV, so the bots should remain solvent), though perhaps the market should offer users to automatically make the arbitrage bets themselves so they can benefit from the presence of the latent variables.
I expect to give an update on this later with more formal derivations, likely together with my implementation of latent class markets for Manifold.
Strictly speaking, it’s an empirical question how well this would work.
I could certainly imagine programmatically fitting a latent variable model to prediction market data, assuming that there is a fixed set of outcomes and the users mainly disagree about the latent variable values. However, it seems to me that this would run into massive problems due to e.g. Kelly betting where people don’t update the probabilities all the way to match their beliefs. You could probably attempt to model the participant’s risk aversion etc., but I have a hard time believing that it would work.
Asking for predictions about the parameters of the model would make it more like a method of moments estimator than a maximum likelihood estimator, right? Or maybe not, it seems kind of hard to predict how exactly it would go.
Regardless, I don’t think this is an unreasonable idea to experiment with, but I think there are two challenges:
As mentioned, I’m pretty skeptical of the notion that you can fit LV models to prediction market predictions and have it work well.
If you don’t have LVPM support, the UI for making predictions would probably be even worse than the UI I made.
I think if this was somehow changed to be maximum likelihood, and if those two challenges were removed, then you would get something close to “My new preferred approach”.