I appreciate this post. A list of frames/examples seems like a good way to communicate an unintuitive or easily misunderstood concept.
As far as I can tell, the biggest frame collision happens around the claim like “even if we ‘solve alignment’, we might be screwed”. I don’t think you address it in this post, except perhaps implicitly in points 2 and 3. One interesting framing is Byrnes’s Law of Conservation of Wisdom.
Other assorted responses;
I don’t wholeheartedly endorse such critiques because I often feel unsure of what exactly people are criticizing when they critique capitalism in this way.
Eh, yeah, “capitalism” has generally been a very common beating boy on the left for the past few decades (e.g., for every mainstream social ill, you’ll probably easily find a bunch of people blaming “capitalism” for it loudly on the internet), and a conflationary alliance term. While there are important and valid variants of some of those critiques, the noise makes it difficult for them to propagate to the “collective consciousness”, on top of them often being difficult to propagate ex ante.[1]
meta-crisis
ETA: Based on a pushback in a private convo from Kaarel, I no longer stand strongly behind the following as pertaining to the concept of the meta-crisis itself. Plausibly, my judgment here was somewhat colored by bad experience with some cluster of enthusiasts of the concept. As far as I recall, I had some more substantive object-level disagreements about this that I can’t recall now what they were, so I am somewhat deferring to my past self by being somewhat skeptical about this concept anyway until I engage further.
[Epistemic status: rant, partially hoping that someone shows me that I am significantly wrong here, at least partly.]
As far as I can tell, from my limited but earnest engagement with this cluster, whenever I try to steer someone towards telling me what exactly those roots of the meta-crisis are, it boils down to bad decision-making, lack of coherent collective agency, etc. All of those things are definitely great to improve, but then calling it a “meta-crisis” doesn’t strike me as a good framing, because it’s like calling a general and broad skill issue a “(meta-)crisis”. One response would be that it’s about the various structures holding up the edifice of civilization collapsing/rotting/decaying, and those respective dynamics of collapse/rot/decay reinforcing one another (I mean, this also happens in the life of an individual human, but OK, let’s acknowledge that the scale of the problem matters). When I made the comment that such things have been occurring for as long as civilizations have been falling, so meta-crisis is not as new a thing as it’s made out to be (except maybe insofar as we have very different sorts of social structures that were not present for the Persians, Maya, Romans, etc.).
I see some ways out here and ways to make the framing more coherent and valuable, at least on the margin, but it doesn’t seem to me like people who are into the meta-crisis framing are trying to make it coherent. Instead, it seems to revolve around a general feeling of “things are getting worse” + pointers to examples of categories of things getting worse and how those ways things are getting worse are interconnected. This makes me suspicious and somewhat uninterested.
But maybe I talked to the wrong people. Maybe the concept of a meta-crisis got corrupted when being transmitted from up high. I welcome links to better explanations of the concept/frame.
</rant>
They also increasingly make more and more aspects of life subject to measurement and control via optimization of metrics, which necessarily fail to capture everything that matters.
I would actually like to see much more unpacked discussion on this, both because of more mundane concerns (e.g., the Acemoglu et al. paper that “failed [your] smell test”; I haven’t read, but it’s very unobvious to me that their concerns are unjustified; I’d probably conclude they’re overly confident in their models, but my model of this has a lot of degrees of freedom, so IDK sounds plausible?), as well as the possibility that, insofar as people are trying to “automate ‘alignment research’”, the automation orchestrators getting WALL-E’d into cognitive enfeeblement and thus approving research directions and solutions that will turn out to be slop.
I appreciate this post. A list of frames/examples seems like a good way to communicate an unintuitive or easily misunderstood concept.
As far as I can tell, the biggest frame collision happens around the claim like “even if we ‘solve alignment’, we might be screwed”. I don’t think you address it in this post, except perhaps implicitly in points 2 and 3. One interesting framing is Byrnes’s Law of Conservation of Wisdom.
Other assorted responses;
Eh, yeah, “capitalism” has generally been a very common beating boy on the left for the past few decades (e.g., for every mainstream social ill, you’ll probably easily find a bunch of people blaming “capitalism” for it loudly on the internet), and a conflationary alliance term. While there are important and valid variants of some of those critiques, the noise makes it difficult for them to propagate to the “collective consciousness”, on top of them often being difficult to propagate ex ante.[1]
ETA: Based on a pushback in a private convo from Kaarel, I no longer stand strongly behind the following as pertaining to the concept of the meta-crisis itself. Plausibly, my judgment here was somewhat colored by bad experience with some cluster of enthusiasts of the concept. As far as I recall, I had some more substantive object-level disagreements about this that I can’t recall now what they were, so I am somewhat deferring to my past self by being somewhat skeptical about this concept anyway until I engage further.
[Epistemic status: rant, partially hoping that someone shows me that I am significantly wrong here, at least partly.]
As far as I can tell, from my limited but earnest engagement with this cluster, whenever I try to steer someone towards telling me what exactly those roots of the meta-crisis are, it boils down to bad decision-making, lack of coherent collective agency, etc. All of those things are definitely great to improve, but then calling it a “meta-crisis” doesn’t strike me as a good framing, because it’s like calling a general and broad skill issue a “(meta-)crisis”. One response would be that it’s about the various structures holding up the edifice of civilization collapsing/rotting/decaying, and those respective dynamics of collapse/rot/decay reinforcing one another (I mean, this also happens in the life of an individual human, but OK, let’s acknowledge that the scale of the problem matters). When I made the comment that such things have been occurring for as long as civilizations have been falling, so meta-crisis is not as new a thing as it’s made out to be (except maybe insofar as we have very different sorts of social structures that were not present for the Persians, Maya, Romans, etc.).
I see some ways out here and ways to make the framing more coherent and valuable, at least on the margin, but it doesn’t seem to me like people who are into the meta-crisis framing are trying to make it coherent. Instead, it seems to revolve around a general feeling of “things are getting worse” + pointers to examples of categories of things getting worse and how those ways things are getting worse are interconnected. This makes me suspicious and somewhat uninterested.
But maybe I talked to the wrong people. Maybe the concept of a meta-crisis got corrupted when being transmitted from up high. I welcome links to better explanations of the concept/frame.
</rant>
See C Thi Nguyen’s Value Capture.
I would actually like to see much more unpacked discussion on this, both because of more mundane concerns (e.g., the Acemoglu et al. paper that “failed [your] smell test”; I haven’t read, but it’s very unobvious to me that their concerns are unjustified; I’d probably conclude they’re overly confident in their models, but my model of this has a lot of degrees of freedom, so IDK sounds plausible?), as well as the possibility that, insofar as people are trying to “automate ‘alignment research’”, the automation orchestrators getting WALL-E’d into cognitive enfeeblement and thus approving research directions and solutions that will turn out to be slop.
Plausibly, a similar situation holds for some “conspiracy theories”.
Good point RE deskilling of alignment researchers.