Yes, now that I think about it, I guess their formalism tends towards incredibly low-signal environments where the actions are primarily simple “tokens” that can be named suggestively but aren’t capable of actually revealing the data needed for the kind of sophistication I’m thinking of. That is, The environment is generally incapable of displaying an environmental tag that would suggest “novel action X (unlike novel actions Y or Z) could be dramatic and irreversible”.
The only way to acquire such insight in a totally “from scratch” game context is to gain experience of having “died” after choosing X (probably several times), or else by having substantially richer environment cues than is normal for systems like this, where concepts like “reversibility” and “predictors of payoff size” could be worked out in trivial contexts and then correctly applied to more significant contexts later on, based on environmental cues that allow the model-based inference of both potential irreversibility and great importance in moderately novel situations.
Yes, now that I think about it, I guess their formalism tends towards incredibly low-signal environments where the actions are primarily simple “tokens” that can be named suggestively but aren’t capable of actually revealing the data needed for the kind of sophistication I’m thinking of. That is, The environment is generally incapable of displaying an environmental tag that would suggest “novel action X (unlike novel actions Y or Z) could be dramatic and irreversible”.
The only way to acquire such insight in a totally “from scratch” game context is to gain experience of having “died” after choosing X (probably several times), or else by having substantially richer environment cues than is normal for systems like this, where concepts like “reversibility” and “predictors of payoff size” could be worked out in trivial contexts and then correctly applied to more significant contexts later on, based on environmental cues that allow the model-based inference of both potential irreversibility and great importance in moderately novel situations.