Except that Zvi coveredthis potential evidence for misalignment and I had this to add. As for the AIs being alien underneath due to training and architecture, I and Claude Opus 4.5 came up with both a case for it and a case against it.
I think your prompt to Claude is pretty leading[1]. You are assuming the answer with “the AIs end up with motivations similar to those of the humans”. The point is that we don’t actually know what their underlying motivations are—we only see how they act when trained and system-prompted into mimicking humans. And no alignment techniques are even 3 9s reliable (and we need >13 9s in the limit of ASI).
Also “Can this crux be partially resolved by, say, studying the values of humans whose brain was developed abnormally” is not thinking at the right level of abstraction. Humans who’s brains developed abnormally are still very close to normal humans in the grand scheme of mindspace. AIs share zero evolutionary history and development (evo-devo), and close to zero brain architecture with humans. Sharing our corpus of media is a very shallow and brittle substitute (i.e. it can make a half-decent mask for the shoggoth, but it doesn’t do anything in the way of evolving the shoggoth into a digital human).
Except that Zvi covered this potential evidence for misalignment and I had this to add. As for the AIs being alien underneath due to training and architecture, I and Claude Opus 4.5 came up with both a case for it and a case against it.
I think your prompt to Claude is pretty leading[1]. You are assuming the answer with “the AIs end up with motivations similar to those of the humans”. The point is that we don’t actually know what their underlying motivations are—we only see how they act when trained and system-prompted into mimicking humans. And no alignment techniques are even 3 9s reliable (and we need >13 9s in the limit of ASI).
Also “Can this crux be partially resolved by, say, studying the values of humans whose brain was developed abnormally” is not thinking at the right level of abstraction. Humans who’s brains developed abnormally are still very close to normal humans in the grand scheme of mindspace. AIs share zero evolutionary history and development (evo-devo), and close to zero brain architecture with humans. Sharing our corpus of media is a very shallow and brittle substitute (i.e. it can make a half-decent mask for the shoggoth, but it doesn’t do anything in the way of evolving the shoggoth into a digital human).
not to mention that the fact that you are using Claude as a trusted source of information on this in the first place is problematic.