…there is no reason to expect optimal score-maxxing play to look anything like play that optimizes for sequence-of-stages completion … This is true in CoastRunners, and in many arcade-style games, and also in, like… Mario, right? It’s not exactly some obscure phenomenon known only to hardcore gamers.
At the time, the talk of the town was the DeepMind Atari-playing DQN (2013, 2015), which almost used pure Atari game score as reward, except that (1) they clipped it to (+1 for any increase, −1 for any decrease, or 0), and (2) they reset the training after one death even if the game allowed multiple lives. According to some LLM I asked just now, the 2015 version fully beat Pong and Boxing, and progressed on a number of other games without getting to the end, but it also found some weird repetitive point-farming strategy in Kangaroo.
At the time, the talk of the town was the DeepMind Atari-playing DQN (2013, 2015), which almost used pure Atari game score as reward, except that (1) they clipped it to (+1 for any increase, −1 for any decrease, or 0), and (2) they reset the training after one death even if the game allowed multiple lives. According to some LLM I asked just now, the 2015 version fully beat Pong and Boxing, and progressed on a number of other games without getting to the end, but it also found some weird repetitive point-farming strategy in Kangaroo.