I think you’re correctly identifying important issues and cracks in the standard ontology, but I think you’re throwing out too much baby in an effort to get rid of bathwater.
For example, I do not think it’s obvious that “Just like vitalistic force, ‘wanting’ is conceptualized as being acausal, i.e. an intrinsic property of an entity with no upstream cause.” In control theory, we can say that a system controls for a thing based on a small collection of mathematical relationships—pressuring an error signal towards zero. While the concept of wanting is overloaded and more complex, I think it makes sense to recognize that “X is controlling for Y” is a valid underpinning that has no vitalistic magic. We can ask what led X to control for Y, or how X controls for Y in terms that are closer to the underlying physics; there’s nothing acausal or intrinsic (except for the definitions, I suppose).
It’s possible for an intuitive human concept X to be a bundle of connotations that don’t add up to anything coherent, while ALSO there’s some similar concept Y that is mathematically well-defined and is capturing many (but not all) of those connotations. Then we can argue about whether X is “really” an imperfect pointer to Y, versus whether X and Y are different but related. But that’s a pointless argument with no answer.
(Fun example: Dan Dennett wrote a book advocating for free will compatibilism in 1984, and then in 2015 added a new preface saying: well actually on second thought, maybe I should have just said all along that we should abandon the term “free will” altogether.)
Anyway, I stand by my claim that acausality is an aspect of how most people intuitively think about wanting. Here’s an example … it’s possible that this tweet is bad-faith, but regardless, I think Marc wouldn’t have said it if it didn’t have some intuitive appeal:
So anyway, we can say that our intuitions around “wanting” have that incoherent aspect (per §2), and we can simultaneously ALSO say that there are well-defined notions of optimization (e.g. in §3.4 I cite Alex Flint’s) that overlap many aspects of the “wanting” intuition, just like you say. Those aren’t contradictory. That’s the whole thing I was trying to do in §3–§4.
Fair enough. And I certainly agree that there is a lot of bathwater! The bundle of connotations attached to the word “wanting” is a mess. I just want to flag that it seems to me that much of the normal ontology can be rescued, albeit with a little bit of work. I claim that concepts like corrigibility are still useful and coherent once the rescuing has taken place.
I think the ontology Steven describes makes a lot of sense for an organism in any environment that contain rocks and plants (easy to predict), weather patterns and running water (hard to predict short term, Vingean approaches of identifying their goals are not very helpful), and other humans and sentient animals (hard to predict short term, but Vingean approaches of identifying their goals are very helpful). This is a good heuristic for an evolved animal to recognize other evolved animals. So of course we have special-purpose learning modules designed for it, and can make this judgement without effort. In fact, since it’s an important part of our threat monitoring systems, our special-purpose abilities in this area are tuned towards avoiding false negatives and tolerating some false positives.
I think you’re correctly identifying important issues and cracks in the standard ontology, but I think you’re throwing out too much baby in an effort to get rid of bathwater.
For example, I do not think it’s obvious that “Just like vitalistic force, ‘wanting’ is conceptualized as being acausal, i.e. an intrinsic property of an entity with no upstream cause.” In control theory, we can say that a system controls for a thing based on a small collection of mathematical relationships—pressuring an error signal towards zero. While the concept of wanting is overloaded and more complex, I think it makes sense to recognize that “X is controlling for Y” is a valid underpinning that has no vitalistic magic. We can ask what led X to control for Y, or how X controls for Y in terms that are closer to the underlying physics; there’s nothing acausal or intrinsic (except for the definitions, I suppose).
It’s possible for an intuitive human concept X to be a bundle of connotations that don’t add up to anything coherent, while ALSO there’s some similar concept Y that is mathematically well-defined and is capturing many (but not all) of those connotations. Then we can argue about whether X is “really” an imperfect pointer to Y, versus whether X and Y are different but related. But that’s a pointless argument with no answer.
(Fun example: Dan Dennett wrote a book advocating for free will compatibilism in 1984, and then in 2015 added a new preface saying: well actually on second thought, maybe I should have just said all along that we should abandon the term “free will” altogether.)
Anyway, I stand by my claim that acausality is an aspect of how most people intuitively think about wanting. Here’s an example … it’s possible that this tweet is bad-faith, but regardless, I think Marc wouldn’t have said it if it didn’t have some intuitive appeal:
So anyway, we can say that our intuitions around “wanting” have that incoherent aspect (per §2), and we can simultaneously ALSO say that there are well-defined notions of optimization (e.g. in §3.4 I cite Alex Flint’s) that overlap many aspects of the “wanting” intuition, just like you say. Those aren’t contradictory. That’s the whole thing I was trying to do in §3–§4.
Fair enough. And I certainly agree that there is a lot of bathwater! The bundle of connotations attached to the word “wanting” is a mess. I just want to flag that it seems to me that much of the normal ontology can be rescued, albeit with a little bit of work. I claim that concepts like corrigibility are still useful and coherent once the rescuing has taken place.
I think the ontology Steven describes makes a lot of sense for an organism in any environment that contain rocks and plants (easy to predict), weather patterns and running water (hard to predict short term, Vingean approaches of identifying their goals are not very helpful), and other humans and sentient animals (hard to predict short term, but Vingean approaches of identifying their goals are very helpful). This is a good heuristic for an evolved animal to recognize other evolved animals. So of course we have special-purpose learning modules designed for it, and can make this judgement without effort. In fact, since it’s an important part of our threat monitoring systems, our special-purpose abilities in this area are tuned towards avoiding false negatives and tolerating some false positives.