Looking at that map of representations of the DNQ agent playing Space Invaders I can’t help thinking if it really has learned any concepts that are similar to what a human would learn. Does the DNQ agent have the concepts of ship, shield, bullet or space invader? Does it have anything that corresponds to the concept of a group of space invaders? Can it generalize? I’m sure human players could quickly adapt if we changed the game so that the ship would shoot from the top to the bottom instead. Does the DNQ agent have anything analogous to an inner simulator? If we showed it a movie where the ship would fly up to the invaders and collide what would it predict happens next?
My gut feeling is that artificial agents are still far away from having reusable and generalizable concepts. It’s one thing, although an impressive one, to use the same framework with identical parameters for different DNQ agents learning different games than it is to use one framework for one agent that learns to play all the games and abstract concepts across them.
This only applies because switching to a QM model is computationally prohibitive. QM is generally held to be more true than CM and even if you’re trying to optimize for things in terms of CM you’re still better off using the QM model as long as you have a good mapping from your QM model to your CM goals.
Humans do indeed find it difficult to think in terms of QM, but this need not be the case for a future AI with access to a quantum computer. If the CM model and the QM model could be run with similar efficiencies then the real issue becomes the mapping from QM model to CM goals. All maps from QM to CM leak in terms of what counts as being located inside the box so the AI might find ways to act outside the box (according to a different mapping). This highlights the point that with computational resources being equal the AI will always prefer the most general available world model for decision making even if its goals are defined in terms a less general model.
I have to point out that the issue with QM to CM mappings is mostly of theoretical interest and in practice it should be possible to define a mapping that safely maximizes the probability of the AI staying in the box while still being able to function optimally. The latter condition is required because a mapping from QM to CM that purely maximizes the probability of the AI staying in box will cause the AI to move in to the middle of the box and cool down.