A piece of the context around this is that I was taught that Reinforcement Learning was a category and not a framework because I never did have a teacher who opened with the non-multiple-choice version of “Supervised” | “Unsupervised” | “RL”—and that messed up the way I think in a way that did literally take years to unlearn. Now the way I think about things isn’t just more aesthetically nicer to me, it’s actually less stressful to think about and feels like it takes up less space and I’m just more effective thinking through things with it. The connections to dynamic programming have gone from “what’s what got to do with this?” to “Wait no it could literally not be any other way.”
This made me angry at first. Because it felt like I was deprived of something. And I looked it up and no indeed I am not the first to have reframed RL as “iterated classification where you set up your classifiers a specific way to nudge your trajectory closer to the goal”. But it taught me something that yesterday’s me really needed to be reminded of. And it gave me a serious degree of confidence that I wouldn’t have had otherwise, about the duality of perspectives that things can take. From one perspective, you have an agent taking actions in a discrete (that part always annoyed me) state-space. In another perspective you have a current which is getting shaped by experiences flowing towards the goal with the sub-currents and other streams becoming wider channels when they connect to the goals. Like the way ants reinforce pathways to the food source.
What fraction of the RL story is actually explained by the equations alone? Is it even half? Can we even give it a quarter? 1/8th? I think I currently give it 1/32th. They told me to stare at the equations more, but when I asked “but what does it mean?” they said “shut up and calculate.” Maybe the picture was in their equations but not mine.
A piece of the context around this is that I was taught that Reinforcement Learning was a category and not a framework because I never did have a teacher who opened with the non-multiple-choice version of “Supervised” | “Unsupervised” | “RL”—and that messed up the way I think in a way that did literally take years to unlearn. Now the way I think about things isn’t just more aesthetically nicer to me, it’s actually less stressful to think about and feels like it takes up less space and I’m just more effective thinking through things with it. The connections to dynamic programming have gone from “what’s what got to do with this?” to “Wait no it could literally not be any other way.”
This made me angry at first. Because it felt like I was deprived of something. And I looked it up and no indeed I am not the first to have reframed RL as “iterated classification where you set up your classifiers a specific way to nudge your trajectory closer to the goal”. But it taught me something that yesterday’s me really needed to be reminded of. And it gave me a serious degree of confidence that I wouldn’t have had otherwise, about the duality of perspectives that things can take. From one perspective, you have an agent taking actions in a discrete (that part always annoyed me) state-space. In another perspective you have a current which is getting shaped by experiences flowing towards the goal with the sub-currents and other streams becoming wider channels when they connect to the goals. Like the way ants reinforce pathways to the food source.
What fraction of the RL story is actually explained by the equations alone? Is it even half? Can we even give it a quarter? 1/8th? I think I currently give it 1/32th. They told me to stare at the equations more, but when I asked “but what does it mean?” they said “shut up and calculate.” Maybe the picture was in their equations but not mine.