It’s a fair point. I don’t think prediction market profit is the best eval of these methods. Pure forecast accuracy is easy to measure, either in live forecasting or past-casting. But I agree prediction market signal is one type of eval, so we do use it at FutureSearch to learn from our mistakes and improve.
Our portfolios and our P&L on Kalshi and Polymarket are at https://markets.futuresearch.ai/. It’s mostly synthetic trading, though real trading at Kalshi has 2 weeks of data now.
i am aware that there is this meme of “you think you can predict stuff, why aren’t you making big $$ on it huh?” this is not...particularly helpful nor my intent.
on my first read, it felt to me like the thing you are working on genuinely is pointed at predicting specific events just like those on prediction markets. know there are...funky tactical bits wrt actually interfacing with markets, but it does seem much much more monetizable that way compared to a lot of other analytical work that is more fuzzily targeted
hence on reflection, i think i would refine to “it seems worth trying to monetize because money is nice, and because it is an imperfect empirical flywheel. if monetization underperforms your other calibration metrics, then such may be a signal that your other calibration metrics are lossy, or it may be a signal that your prediction-market-praxis has gaps, and in either case that seems like valuable signals (former → epistemics, latter → money since i would suspect that improving prediction-market-praxis to unblock conversion of predictions into money is an efficient use of effort” money is, after all, quite useful.
and on the one hand, that is super cool and i wish you success in making gobs of money! and on the other hand, if you truly cannot monetize it this way, then i think that is informative to my mental-model of the capability you are honing
It’s a fair point. I don’t think prediction market profit is the best eval of these methods. Pure forecast accuracy is easy to measure, either in live forecasting or past-casting. But I agree prediction market signal is one type of eval, so we do use it at FutureSearch to learn from our mistakes and improve.
Our portfolios and our P&L on Kalshi and Polymarket are at https://markets.futuresearch.ai/. It’s mostly synthetic trading, though real trading at Kalshi has 2 weeks of data now.
i am aware that there is this meme of “you think you can predict stuff, why aren’t you making big $$ on it huh?” this is not...particularly helpful nor my intent.
on my first read, it felt to me like the thing you are working on genuinely is pointed at predicting specific events just like those on prediction markets. know there are...funky tactical bits wrt actually interfacing with markets, but it does seem much much more monetizable that way compared to a lot of other analytical work that is more fuzzily targeted
hence on reflection, i think i would refine to “it seems worth trying to monetize because money is nice, and because it is an imperfect empirical flywheel. if monetization underperforms your other calibration metrics, then such may be a signal that your other calibration metrics are lossy, or it may be a signal that your prediction-market-praxis has gaps, and in either case that seems like valuable signals (former → epistemics, latter → money since i would suspect that improving prediction-market-praxis to unblock conversion of predictions into money is an efficient use of effort” money is, after all, quite useful.
and on the one hand, that is super cool and i wish you success in making gobs of money! and on the other hand, if you truly cannot monetize it this way, then i think that is informative to my mental-model of the capability you are honing
Agree, whether a world-modeling technique makes money does seem like a load bearing part of one’s mental model of it.
I do want to emphasize though, directly testing forecast accuracy via Brier scores is often easier and more informative.