I found this post very unintuitive, and I’m not sure Approval Reward is a precisely bounded concept. If it can be used to explain “saving money to buy a car” then it can really be stretched to explain a wide range of human actions that IMO can be better explained by drives other than social approval. Most importantly (and this is likely a skill issue on my part) it’s unclear to me how to operationalize what would be predicted by an Approval Reward driven framework vs some alternative.
What alternative? I assume you’re not proposing that people aren’t motivated by approval? The field of behavioral neuroscience and other branches of psychology all take it as pretty much a given that animals including humans are motivated by social reward. There’s some chance this is all wrong, or more plausibly, that social reward sits downstream of noticing that you get food and shelter after noticing social reward. But probably it’s a built-in base drive. The relevant question is how much we’re motivated by it, and how often. Humans have a bunch of different motivating factors. Steve is arguing that social reward is the most relevant one for most people most of the time, and I think that’s right.
Approval Reward (AR) is a particular kind of corrigibility, so anything that isn’t corrigibility isn’t AR and some things that are corrigibility still aren’t AR. The concept is bounded. Although, precise bounding doesn’t seem valuable while first exploring concepts. First come the fuzzy bounds, then the bounds can be shored up where it is important.
I agree with you that people are complex and AR probably doesn’t apply to all instances of someone saving to buy a car, but I’d be surprised if AR never applied to someone saving to buy a car.
The most important prediction of AR is that people misgeneralizing AR onto non-human entities make mistakes in predicting those non-human entities. People who approach wild animals are an example of this. I think anthropomorphizing machines is possible. I bet people would be less safety conscious around an industrial machine made to look friendly than one made to look scary. And most importantly, what I see as the key point of this post, we can predict that people who are optimistic about AGI are more likely to be reasoning based on AR as opposed to reasoning based on systems dynamics and theory of agents.
My own internal gears level model suggests that AR is only one component of what makes people erroneously optimistic. Other possibilities being that people are too optimistic by default, that people are incentivize to be optimistic, and the abundance of salient (but imprecise) reference classes with examples suggesting things will go well. In fact, AR could be seen as an important specific case of a reference class.
I found this post very unintuitive, and I’m not sure Approval Reward is a precisely bounded concept. If it can be used to explain “saving money to buy a car” then it can really be stretched to explain a wide range of human actions that IMO can be better explained by drives other than social approval. Most importantly (and this is likely a skill issue on my part) it’s unclear to me how to operationalize what would be predicted by an Approval Reward driven framework vs some alternative.
What alternative? I assume you’re not proposing that people aren’t motivated by approval? The field of behavioral neuroscience and other branches of psychology all take it as pretty much a given that animals including humans are motivated by social reward. There’s some chance this is all wrong, or more plausibly, that social reward sits downstream of noticing that you get food and shelter after noticing social reward. But probably it’s a built-in base drive. The relevant question is how much we’re motivated by it, and how often. Humans have a bunch of different motivating factors. Steve is arguing that social reward is the most relevant one for most people most of the time, and I think that’s right.
Approval Reward (AR) is a particular kind of corrigibility, so anything that isn’t corrigibility isn’t AR and some things that are corrigibility still aren’t AR. The concept is bounded. Although, precise bounding doesn’t seem valuable while first exploring concepts. First come the fuzzy bounds, then the bounds can be shored up where it is important.
I agree with you that people are complex and AR probably doesn’t apply to all instances of someone saving to buy a car, but I’d be surprised if AR never applied to someone saving to buy a car.
The most important prediction of AR is that people misgeneralizing AR onto non-human entities make mistakes in predicting those non-human entities. People who approach wild animals are an example of this. I think anthropomorphizing machines is possible. I bet people would be less safety conscious around an industrial machine made to look friendly than one made to look scary. And most importantly, what I see as the key point of this post, we can predict that people who are optimistic about AGI are more likely to be reasoning based on AR as opposed to reasoning based on systems dynamics and theory of agents.
My own internal gears level model suggests that AR is only one component of what makes people erroneously optimistic. Other possibilities being that people are too optimistic by default, that people are incentivize to be optimistic, and the abundance of salient (but imprecise) reference classes with examples suggesting things will go well. In fact, AR could be seen as an important specific case of a reference class.