When you encounter a particular state, you only update the Q-value of that state in the table, and don’t do anything to the Q-values of any other state. Therefore, seeing one state will make no difference to your policy on any other state, i.e. no generalization.
You need to use function approximators of some sort to see generalization to new states. (This doesn’t have to be a neural net—you could approximate the Q-function as a linear function over handcoded features, and this would also give you some generalization to new states.)
Okay, thanks for the explanation!
Yeah, I was ignoring the “all possible RL methods” because it was vague (and I also expect it not to work for any specific formalization, e.g. you’d have to rule out RL methods that say “if the goal is G, then output <specific policy>, otherwise do regular RL”, which seems non-trivial). If you use only a few RL methods, then I think I would stick with my claim about generalization:
Yes, trying to ensure that not all policies are generated is indeed the main issue here. It also underlies the resource condition. This makes me think that maybe using RL is not the appropriate way. That being said, I still think an approach exists for computing focus instead of competence. I just don’t know it yet.
I could add the sentence “Alternatively, if you try to use “all possible” RL algorithms, I expect that there will be many pathological RL algorithms that effectively make any policy goal-directed”, if you wanted me to, but I think the version with a small set of RL algorithms seems better to me and I’d rather keep the focus on that.
I agree that keeping the focus (!) on the more realistic case makes more sense here.
Okay, thanks for the explanation!
Yes, trying to ensure that not all policies are generated is indeed the main issue here. It also underlies the resource condition. This makes me think that maybe using RL is not the appropriate way. That being said, I still think an approach exists for computing focus instead of competence. I just don’t know it yet.
I agree that keeping the focus (!) on the more realistic case makes more sense here.