i think the survey respondents here are talking about something meaningful and i probably agree with most of their judgments about that thing. for example, with that notion of coherence, i probably agree with “Google (the company) is less coherent now than it was when it had <10 employees” (and this is so even though Google is more capable now than it was when it had 10 employees)
I agree, but there’s a caveat that the notion of coherence as operationalized in the linked Sohl-Dickstein post conflates at least two (plausibly more) notions. The three questions he uses to point at the concept of “coherence” are:
How well can the entity’s behavior be explained as trying to optimize a single fixed utility function?
How well aligned is the entity’s behavior with a coherent and self-consistent set of goals?
To what degree is the entity not a hot mess of self-undermining behavior?
I expect the first two (in most of the respondents) to (connotationally/associationally) evoke the image of an entity/agent that wants some relatively specific and well-defined thing, and this is largely why you get the result that a thermostat is more of a “coherent agent” than Google. But then this just says that with more intelligence, you are capable of reasonably skillfully pursuing more complicated, convoluted, and [not necessarily that related to each other] goals/values, which is not surprising. Another part is that real-world intelligent agents (those capable of ~learning), at least to some extent, do some sort of figuring out / constructing their actual values on the fly or change their mind about what they value.
The third question is pointing at something different: being composed of purposive forces pushing the world in different directions. Metaphorically, something like destructive constructive interference vs destructive interference or channeling energy to do useful work vs that energy dissipating as waste heat. Poetically, each purposive part has a seed of some purpose in it, and when they compose in the right way, there’s “superadditivity” of those purposive parts: they add up to a big effect consistent with the purposes of the purposive parts. “Composition preserves purpose.”
A clear human example of incoherence in this sense is someone repeating a cycle of (1) making a specific sort of commitment; and then (2) deciding to abandon that commitment, and continuing to repeat this cycle, even though “they should notice” that the track record clearly indicates they’re not going to follow through on this commitment, so they should change something about how they approach the goal the commitment is instrumental for. In this example, the parts of the agent that [fail to cohere]/[push the world in antagonistic directions] are some of their constitutive agent episodes across time.
One vague picture here is that the pieces of the mind are trying to compose in a way that achieves some “big effect”.
Your example of superficially learning some area of math for algebraic crunching, but without deep understanding and integration with the rest of your mathematical knowledge, is an example of something “less bad”, which we might call “unfulfilled positive interference”. The new piece of math does not “actively discohere”, because it doesn’t screw up your prior understanding. But there might be some potential for further synergy that is not being fulfilled until you integrate it.
To sum up, a highly coherent agent in this sense may have very convoluted values, and so Sohl-Dickstein’s “coherence question 3” diverges from “coherence questions 1 and 2″.
But then there’s a further thing. Being incoherent can be “fine” if you are sufficiently intelligent to handle it. Or maybe more to the point, your capacities suffice to control/bound/limit the damage/loss that your incoherence incurs. You have a limited amount of “optimization power” and could spend some of it on coherentizing yourself, but you figure that you’re going to gain more of what you want if you spend that optimization power on doing what you want to do with the somewhat incoherent structures that you have already (or you cohere yourself a bit, but not as much as you might).[1] E.g., you can have agents A and B, such that A is more intelligent than B, and A is less coherent than B, but the difference in intelligence is sufficient for A to just permanently disempower B. A could self-coherentize more, but doesn’t have to.
It would be interesting (I am just injecting this hypothesis into the hypothesis space without claiming I have (legible) evidence for it) if it turned out that, given some mature way of measuring intelligence and coherence, relatively small marginal gains in intelligence often offset relatively large losses in coherence in terms of something like “general capacity to effectively pursue X class of goals”.
With the caveat to this that the more maximizery/unbounded the values are, the more the goal-optimal optimization power allocation shifts towards actually frontloading a lot of self-coherentizing as capital investment.
i think you’re right that the sohl-dickstein post+survey also conflates different notions, and i might even have added more notions into the mix with my list of questions trying to get at some notion(s)[1]
a monograph untangling this coherence mess some more would be valuable. it could do the following things:
specifying a bunch of a priori different properties that could be called “coherence”
discussing which ones are equivalent, which ones are correlated, which ones seem pretty independent
giving good names to the notions or notion-clusters
discussing which kinds of coherence generically increase/decrease with capabilities, which ones probably increase/decrease with capabilities in practice, which ones can both increase or decrease with capabilities depending on the development/learning process, both around human level and later/eventually, in human-like minds and more generally[2]
discussing how this relates to AI x risk. like, which kinds of coherence should play a role in a case for AI x risk? what does that look like? or maybe the picture should make one optimistic about some approach to de-AGI-x-risk-ing? or about AGI in general?[3]
the answers to some of these questions might depend on some partly “metaphysical” facts like whether math is genuinely infinite or whether technological maturity is a thing
Probably not a full-monograph-length monograph, because I don’t think either that (1) the coherence-related confusions are isolated from other confused concepts in this line of inquiry, or that (2) the descendants of the concept of “coherence” will be related in some “nature-at-joint-carving” way, which would justify discussing them jointly. (Those are the two reasons I see why we might want a full-monograph-length monograph untangling the mess of some specific, confused concept.)
But an investigation (of TBD length) covering at least the first three of your bullet points seems good. I’m less sure about the latter two, probably because I expect that after the first three steps, a lot of new salient questions will appear, whereas the then-available answers to the relationship with capabilities will be rather scant (plausibly because the concept of capabilities itself would need to be refactored for more answers to be available), and that just the result of this single-concept-deconfusing investigation will have rather little implications for AGI X-risk (but might be a fruitful input to future investigation, which is the point).
I agree, but there’s a caveat that the notion of coherence as operationalized in the linked Sohl-Dickstein post conflates at least two (plausibly more) notions. The three questions he uses to point at the concept of “coherence” are:
How well can the entity’s behavior be explained as trying to optimize a single fixed utility function?
How well aligned is the entity’s behavior with a coherent and self-consistent set of goals?
To what degree is the entity not a hot mess of self-undermining behavior?
I expect the first two (in most of the respondents) to (connotationally/associationally) evoke the image of an entity/agent that wants some relatively specific and well-defined thing, and this is largely why you get the result that a thermostat is more of a “coherent agent” than Google. But then this just says that with more intelligence, you are capable of reasonably skillfully pursuing more complicated, convoluted, and [not necessarily that related to each other] goals/values, which is not surprising. Another part is that real-world intelligent agents (those capable of ~learning), at least to some extent, do some sort of figuring out / constructing their actual values on the fly or change their mind about what they value.
The third question is pointing at something different: being composed of purposive forces pushing the world in different directions. Metaphorically, something like destructive constructive interference vs destructive interference or channeling energy to do useful work vs that energy dissipating as waste heat. Poetically, each purposive part has a seed of some purpose in it, and when they compose in the right way, there’s “superadditivity” of those purposive parts: they add up to a big effect consistent with the purposes of the purposive parts. “Composition preserves purpose.”
A clear human example of incoherence in this sense is someone repeating a cycle of (1) making a specific sort of commitment; and then (2) deciding to abandon that commitment, and continuing to repeat this cycle, even though “they should notice” that the track record clearly indicates they’re not going to follow through on this commitment, so they should change something about how they approach the goal the commitment is instrumental for. In this example, the parts of the agent that [fail to cohere]/[push the world in antagonistic directions] are some of their constitutive agent episodes across time.
One vague picture here is that the pieces of the mind are trying to compose in a way that achieves some “big effect”.
Your example of superficially learning some area of math for algebraic crunching, but without deep understanding and integration with the rest of your mathematical knowledge, is an example of something “less bad”, which we might call “unfulfilled positive interference”. The new piece of math does not “actively discohere”, because it doesn’t screw up your prior understanding. But there might be some potential for further synergy that is not being fulfilled until you integrate it.
To sum up, a highly coherent agent in this sense may have very convoluted values, and so Sohl-Dickstein’s “coherence question 3” diverges from “coherence questions 1 and 2″.
But then there’s a further thing. Being incoherent can be “fine” if you are sufficiently intelligent to handle it. Or maybe more to the point, your capacities suffice to control/bound/limit the damage/loss that your incoherence incurs. You have a limited amount of “optimization power” and could spend some of it on coherentizing yourself, but you figure that you’re going to gain more of what you want if you spend that optimization power on doing what you want to do with the somewhat incoherent structures that you have already (or you cohere yourself a bit, but not as much as you might).[1] E.g., you can have agents A and B, such that A is more intelligent than B, and A is less coherent than B, but the difference in intelligence is sufficient for A to just permanently disempower B. A could self-coherentize more, but doesn’t have to.
It would be interesting (I am just injecting this hypothesis into the hypothesis space without claiming I have (legible) evidence for it) if it turned out that, given some mature way of measuring intelligence and coherence, relatively small marginal gains in intelligence often offset relatively large losses in coherence in terms of something like “general capacity to effectively pursue X class of goals”.
With the caveat to this that the more maximizery/unbounded the values are, the more the goal-optimal optimization power allocation shifts towards actually frontloading a lot of self-coherentizing as capital investment.
i think you’re right that the sohl-dickstein post+survey also conflates different notions, and i might even have added more notions into the mix with my list of questions trying to get at some notion(s) [1]
a monograph untangling this coherence mess some more would be valuable. it could do the following things:
specifying a bunch of a priori different properties that could be called “coherence”
discussing which ones are equivalent, which ones are correlated, which ones seem pretty independent
giving good names to the notions or notion-clusters
discussing which kinds of coherence generically increase/decrease with capabilities, which ones probably increase/decrease with capabilities in practice, which ones can both increase or decrease with capabilities depending on the development/learning process, both around human level and later/eventually, in human-like minds and more generally [2]
discussing how this relates to AI x risk. like, which kinds of coherence should play a role in a case for AI x risk? what does that look like? or maybe the picture should make one optimistic about some approach to de-AGI-x-risk-ing? or about AGI in general? [3]
i didn’t re-read that post before writing my comment above
the answers to some of these questions might depend on some partly “metaphysical” facts like whether math is genuinely infinite or whether technological maturity is a thing
i think the optimistic conclusions are unlikely, but i wouldn’t want to pre-write that conclusion for the monograph, especially if i’m not writing it
Yeah.
Probably not a full-monograph-length monograph, because I don’t think either that (1) the coherence-related confusions are isolated from other confused concepts in this line of inquiry, or that (2) the descendants of the concept of “coherence” will be related in some “nature-at-joint-carving” way, which would justify discussing them jointly. (Those are the two reasons I see why we might want a full-monograph-length monograph untangling the mess of some specific, confused concept.)
But an investigation (of TBD length) covering at least the first three of your bullet points seems good. I’m less sure about the latter two, probably because I expect that after the first three steps, a lot of new salient questions will appear, whereas the then-available answers to the relationship with capabilities will be rather scant (plausibly because the concept of capabilities itself would need to be refactored for more answers to be available), and that just the result of this single-concept-deconfusing investigation will have rather little implications for AGI X-risk (but might be a fruitful input to future investigation, which is the point).