The scoring for that first game is downright bizarre. The optimal strategy for picking probabilities does not reflect the actual relative likelihoods of the options, but says “don’t overthink it”. In order to do well, you must overthink it.
(I run the team that created that game. I made the guess-most-likely-next-token game and Fabien Roger made the other one.)
The optimal strategy for picking probabilities in that game is to say what your probability for those two next tokens would have been if you hadn’t updated on being asked about them. What’s your problem with this?
It’s kind of sad that this scoring system is kind of complicated. But I don’t know how to construct simpler games such that we can unbiasedly infer human perplexity from what the humans do.
The scoring for that first game is downright bizarre. The optimal strategy for picking probabilities does not reflect the actual relative likelihoods of the options, but says “don’t overthink it”. In order to do well, you must overthink it.
(I run the team that created that game. I made the guess-most-likely-next-token game and Fabien Roger made the other one.)
The optimal strategy for picking probabilities in that game is to say what your probability for those two next tokens would have been if you hadn’t updated on being asked about them. What’s your problem with this?
It’s kind of sad that this scoring system is kind of complicated. But I don’t know how to construct simpler games such that we can unbiasedly infer human perplexity from what the humans do.
Yeah, if anyone builds a better version of this game, please let me know!