Humans [in contrast with RL-based agents] have heuristics like vindictiveness that help prevent us from being manipulated by extortion, and which seem particularly effective against implicit extortion. Modern humans are also capable of doing explicit reasoning to recognize the costs of giving in to extortion.
What implications does this have for the claim (which I have seen promulgated here on Less Wrong as if it were obviously true and accepted) that humans in fact are reinforcement learners?
I’m not quite sure how to make sense of this reply, and it feels like there is an implication here that I’m not parsing; could you elaborate? Presumably, the idea is that our reward function is indeed “carefully constructed” by evolution. (Note that I’m trying to extrapolate from memory of past discussions; folks who have actually made the “humans are reinforcement learners” claim should please feel free to jump in here!)
If you model a human as an RL agent, then a lot of the work is being done by a very carefully constructed reward function. You can tell since humans do a lot of stuff that an RL agent basically wouldn’t do (like “die for a cause”). You can bake an awful lot into a carefully constructed reward function—for example, you can reward the agent whenever it takes actions that are optimal according to some arbitrary decision theory X—so it’s probably possible to describe a human as an RL agent but it doesn’t seem like a useful description.
At any rate, once the reward function is doing a lot of the optimization, the arguments in this post don’t really apply. Certainly an RL agent can have a heuristic like vindictiveness if you just change the reward function.
What implications does this have for the claim (which I have seen promulgated here on Less Wrong as if it were obviously true and accepted) that humans in fact are reinforcement learners?
That claim is only plausible if you use a very carefully constructed reward function.
I’m not quite sure how to make sense of this reply, and it feels like there is an implication here that I’m not parsing; could you elaborate? Presumably, the idea is that our reward function is indeed “carefully constructed” by evolution. (Note that I’m trying to extrapolate from memory of past discussions; folks who have actually made the “humans are reinforcement learners” claim should please feel free to jump in here!)
If you model a human as an RL agent, then a lot of the work is being done by a very carefully constructed reward function. You can tell since humans do a lot of stuff that an RL agent basically wouldn’t do (like “die for a cause”). You can bake an awful lot into a carefully constructed reward function—for example, you can reward the agent whenever it takes actions that are optimal according to some arbitrary decision theory X—so it’s probably possible to describe a human as an RL agent but it doesn’t seem like a useful description.
At any rate, once the reward function is doing a lot of the optimization, the arguments in this post don’t really apply. Certainly an RL agent can have a heuristic like vindictiveness if you just change the reward function.
That makes sense, thank you.