It seems to me that you can game the system by finding one algorithm that would make you some money. Then keep submitting very slightly different version of the same algorithm (so they’d have the same complexity) over and over, receiving additional money for each one. (If I’m understanding the proposed system correctly.)
Another way is to submit algorithms that are essentially random. If you submit enough of them, some of them will do well enough to earn you money.
Yet another way is to submit an algorithm that essentially encodes the history of the thing you’re trying to predict (i.e. overfit). It seems like it would be maximally rewarded under this system.
My proposal: reward models (and their creators) only based on how well they are predicting incoming data after the model has been submitted. Also submitting an algorithm costs you some money.
It seems to me that you can game the system by finding one algorithm that would make you some money. Then keep submitting very slightly different version of the same algorithm (so they’d have the same complexity) over and over, receiving additional money for each one. (If I’m understanding the proposed system correctly.)
I agree, this is a problem. I think it’s kind of not a problem “in the limit of competitive play”—the first person to discover a theory will do this before everyone else can spam the system with near clones of the theory, so they’ll get the deserved credit.
From a Bayesian perspective, this is just paying people for correctly pointing out that an elegant theory’s probability is more than just the probability of its most elegant formulation; it also gains credibility from other formulations.
Yet another way is to submit an algorithm that essentially encodes the history of the thing you’re trying to predict (i.e. overfit). It seems like it would be maximally rewarded under this system.
My idea was that the prior probability of this would be so low that the reward here is low. It’s like knowing a stock is a sure thing—you know it multiplies in value a million fold—but the stock only has a millionth of a cent available to buy.
Supposed argument: encoding a history of length L (in bits) is going to put you at roughly 2−L. If the market was only able to predict those bits 50-50, then you could climb up to roughly probability 1 from this. But in practice the market should be better than random, so you should do worse.
Things will actually be a bit worse than 2−L because we also have to encode “here’s the literal history:” (since giving the literal history will be an unusual thing in the prior, IE, not something we make costless in the description language).
Now that I’ve written the argument out, though, it isn’t as good as I’d like.
The market will be 50-50 on some things, so there’s non-negligible value to pick up in those cases.
The whole point is that you can sneak predictions in before the market predicts well, so you can get in while things are close to 50-50. My argument therefore depends on restricting this, EG always making the “first price” = the price after the first day of trading, to give the humans a bit of an edge over submitted algorithms.
So, yeah, this seems like a serious problem to think about.
I thought that there was only ever a finite pool of money to be obtained by submitting algorithms, but this is clearly wrong due to this trick. The winnings have to somehow be normalized, but it’s not clear how to do this.
My proposal: reward models (and their creators) only based on how well they are predicting incoming data after the model has been submitted. Also submitting an algorithm costs you some money.
This defeats the whole point—there’s no reason to submit algorithms rather than just bet according to them.
It seems to me that you can game the system by finding one algorithm that would make you some money. Then keep submitting very slightly different version of the same algorithm (so they’d have the same complexity) over and over, receiving additional money for each one. (If I’m understanding the proposed system correctly.)
Another way is to submit algorithms that are essentially random. If you submit enough of them, some of them will do well enough to earn you money.
Yet another way is to submit an algorithm that essentially encodes the history of the thing you’re trying to predict (i.e. overfit). It seems like it would be maximally rewarded under this system.
My proposal: reward models (and their creators) only based on how well they are predicting incoming data after the model has been submitted. Also submitting an algorithm costs you some money.
I agree, this is a problem. I think it’s kind of not a problem “in the limit of competitive play”—the first person to discover a theory will do this before everyone else can spam the system with near clones of the theory, so they’ll get the deserved credit.
From a Bayesian perspective, this is just paying people for correctly pointing out that an elegant theory’s probability is more than just the probability of its most elegant formulation; it also gains credibility from other formulations.
My idea was that the prior probability of this would be so low that the reward here is low. It’s like knowing a stock is a sure thing—you know it multiplies in value a million fold—but the stock only has a millionth of a cent available to buy.
Supposed argument: encoding a history of length L (in bits) is going to put you at roughly 2−L. If the market was only able to predict those bits 50-50, then you could climb up to roughly probability 1 from this. But in practice the market should be better than random, so you should do worse.
Things will actually be a bit worse than 2−L because we also have to encode “here’s the literal history:” (since giving the literal history will be an unusual thing in the prior, IE, not something we make costless in the description language).
Now that I’ve written the argument out, though, it isn’t as good as I’d like.
The market will be 50-50 on some things, so there’s non-negligible value to pick up in those cases.
The whole point is that you can sneak predictions in before the market predicts well, so you can get in while things are close to 50-50. My argument therefore depends on restricting this, EG always making the “first price” = the price after the first day of trading, to give the humans a bit of an edge over submitted algorithms.
So, yeah, this seems like a serious problem to think about.
I thought that there was only ever a finite pool of money to be obtained by submitting algorithms, but this is clearly wrong due to this trick. The winnings have to somehow be normalized, but it’s not clear how to do this.
This defeats the whole point—there’s no reason to submit algorithms rather than just bet according to them.