But if a metric is trivially gameable, surely that makes it sus and less impressive, even if someone is not trivially, or even at all gaming it.
Why would you think that? Surely the reason that a metric being gameable matters is if… someone is or might be gaming it?
Plenty of metrics are gameable in theory, but are still important and valid given that you usually can tell if they are. Apply this to any of the countless measurements you take for granted.
Someone comes to you and say ‘by dint of diet, hard work (and a bit of semaglutide), my bathroom scale says I’ve lost 50 pounds over the past year’. Do you say ‘do you realize how trivially gameable that metric is? how utterly sus and unimpressive? You could have just been holding something the first time, or taken a foot off the scale the second time. Nothing would be easier than to fake this. Does this bathroom scale even exist in the first place?’ Or, ‘my thermometer says I’m running a fever of 105F, I am dying, take me to the hospital right now’ - ‘you gullible fool, do you have any idea how easy that is to manipulate by dunking it in a mug of tea or something? sus. Get me some real evidence before I waste all that time driving you to the ER.’
Hmm yea gameability might not be so interesting of a property of metrics as I’ve expressed.
(though I still feel there is something in there. Fixing your calibration chart after the fact by predicting one-sided coins dice is maybe a lot like taking a foot off the bathroom scale. But, for example, predicting every event as a constant p%, is that even cheating in the calibration game? Though neither of these directly applies to the case of prediction market platforms)
Why would you think that? Surely the reason that a metric being gameable matters is if… someone is or might be gaming it?
Plenty of metrics are gameable in theory, but are still important and valid given that you usually can tell if they are. Apply this to any of the countless measurements you take for granted. Someone comes to you and say ‘by dint of diet, hard work (and a bit of semaglutide), my bathroom scale says I’ve lost 50 pounds over the past year’. Do you say ‘do you realize how trivially gameable that metric is? how utterly sus and unimpressive? You could have just been holding something the first time, or taken a foot off the scale the second time. Nothing would be easier than to fake this. Does this bathroom scale even exist in the first place?’ Or, ‘my thermometer says I’m running a fever of 105F, I am dying, take me to the hospital right now’ - ‘you gullible fool, do you have any idea how easy that is to manipulate by dunking it in a mug of tea or something? sus. Get me some real evidence before I waste all that time driving you to the ER.’
Hmm yea gameability might not be so interesting of a property of metrics as I’ve expressed.
(though I still feel there is something in there. Fixing your calibration chart after the fact by predicting one-sided
coinsdice is maybe a lot like taking a foot off the bathroom scale. But, for example, predicting every event as a constant p%, is that even cheating in the calibration game? Though neither of these directly applies to the case of prediction market platforms)