To save time, boring games such as random environments or one where the agent can ‘die’* are excluded
* make a mistake which turns any future rewards into fixed rewards with no connection to future actions
This seems like a glaring bias in algorithm design… like it is somehow building anthropic craziness into the definition of intelligence on purpose? If I was trying to design something that maximized its score it would never bother to consider situations of possible death because they “don’t count” in the context it is practicing within, where it is presumably deploying priors/habits/heuristics for use in other contexts. Generalizing this insight to other design or educational processes is kind of worry inducing...
‘die’ is my own term, since it seemed to be the game term analogous to ‘when an agent makes a move that renders it causally unconnected to all future rewards’ (again, my own description).
The problem with including games in which one can ‘die’ is that they take much longer to learn. Suppose the agent the first time it plays the game happens to ‘die’, and now it only experiences a steady stream of 1,1,1,1,1… (low rewards). Nothing it does changes its future rewards, so exploration (trying different moves) is penalized. Dying on the first move might look like a good strategy!
Imagine if the rules looked like this: die~>1,1,1,1,...; not-die~>either −1 or +10. If the agent first tried out die, saw the +1 rewards, then the next game chose not-die and got −1, it may permanently start exploring down the die branch. An agent might eventually go back and try the not-die route and finally discover the +10, but this would take a while and is at odds with the idea of a reasonably quickly administered IQ test. Better to exclude such tests and switch to a more complex one.
Yes, now that I think about it, I guess their formalism tends towards incredibly low-signal environments where the actions are primarily simple “tokens” that can be named suggestively but aren’t capable of actually revealing the data needed for the kind of sophistication I’m thinking of. That is, The environment is generally incapable of displaying an environmental tag that would suggest “novel action X (unlike novel actions Y or Z) could be dramatic and irreversible”.
The only way to acquire such insight in a totally “from scratch” game context is to gain experience of having “died” after choosing X (probably several times), or else by having substantially richer environment cues than is normal for systems like this, where concepts like “reversibility” and “predictors of payoff size” could be worked out in trivial contexts and then correctly applied to more significant contexts later on, based on environmental cues that allow the model-based inference of both potential irreversibility and great importance in moderately novel situations.
This seems like a glaring bias in algorithm design… like it is somehow building anthropic craziness into the definition of intelligence on purpose? If I was trying to design something that maximized its score it would never bother to consider situations of possible death because they “don’t count” in the context it is practicing within, where it is presumably deploying priors/habits/heuristics for use in other contexts. Generalizing this insight to other design or educational processes is kind of worry inducing...
‘die’ is my own term, since it seemed to be the game term analogous to ‘when an agent makes a move that renders it causally unconnected to all future rewards’ (again, my own description).
The problem with including games in which one can ‘die’ is that they take much longer to learn. Suppose the agent the first time it plays the game happens to ‘die’, and now it only experiences a steady stream of 1,1,1,1,1… (low rewards). Nothing it does changes its future rewards, so exploration (trying different moves) is penalized. Dying on the first move might look like a good strategy!
Imagine if the rules looked like this: die~>1,1,1,1,...; not-die~>either −1 or +10. If the agent first tried out die, saw the +1 rewards, then the next game chose not-die and got −1, it may permanently start exploring down the die branch. An agent might eventually go back and try the not-die route and finally discover the +10, but this would take a while and is at odds with the idea of a reasonably quickly administered IQ test. Better to exclude such tests and switch to a more complex one.
Yes, now that I think about it, I guess their formalism tends towards incredibly low-signal environments where the actions are primarily simple “tokens” that can be named suggestively but aren’t capable of actually revealing the data needed for the kind of sophistication I’m thinking of. That is, The environment is generally incapable of displaying an environmental tag that would suggest “novel action X (unlike novel actions Y or Z) could be dramatic and irreversible”.
The only way to acquire such insight in a totally “from scratch” game context is to gain experience of having “died” after choosing X (probably several times), or else by having substantially richer environment cues than is normal for systems like this, where concepts like “reversibility” and “predictors of payoff size” could be worked out in trivial contexts and then correctly applied to more significant contexts later on, based on environmental cues that allow the model-based inference of both potential irreversibility and great importance in moderately novel situations.