Great article, but I might be biased since I’m also a fan of Chapman. I find the comments to be fascinating.
It seems to me that people read your article and think, “oh, that’s not new, I have a patch for that in my system.”
However. I think the point you and Chapman are trying to make is that we should think about these patches at the interface of rationality and the real world more carefully. The missing connection, then, is people wondering, why? Are you trying to:
simply argue that the problem exists?
propose a comprehensive approach?
propose a specific instance of the approach? connect stronger to the operations of human hardware?
And, would 3) end up looking itself like a patch?
Also, very curious to hear your thoughts on normativity. That’s usually the subject of ethics, so would the discussion become meta ethical?
Thank you for the thoughtful review, and laying the land!
I work in AI4Science, and have only recently started following the LessWrong thread of AI alignment.
In the spirit of seeking to learn, I wanted to ask: instead of all of these maximalist claims for mindshare, I was wondering if there are more “mundane” predictions, e.g. something like a “proto-ASI” missing a lot of important aspects of ASI will nevertheless be powerful enough to be destructive (“Most people will/die or at least be miserable” instead of “Everyone Dies”) because those who have enough resources to wield it are necessarily not aligned with most people (even without malice, simply by virtue of the argument of the representational capacities of their few brains can not possibly cover the interests of most people). I also recognize that this is under the assumption that the interests of most people are not reducible to, e.g. a simply graspable, strict mono-culture. (e.g. a Handmaid’s Tale or any other similar fantasy world).
And there are so many other examples of human inter-misalignment that I have a hard time imagining that “AI alignment” is even sufficient to avoid really bad scenarios. Another related point is that it’s not obvious to me that human values sufficiently constrain behavior. The simplest example is the value of family; even though most people have this same value as probably one of their top values, families are in different positions and so protecting the family/tribe, leads to different actions because it is applied to different context. Same objective; different boundary conditions; different outcomes. It is insufficient to align on objectives; but we also need the much messier and radical alignment of boundary conditions, which gets into the material constraints of the world.
Relatedly, what are the best estimates on the resources-required to sustain and power/operate an ASI?
Thank you!