Somebody asked “Why believe that?” of “Not more than one millionth.” I suppose it’s a fair question if somebody doesn’t see it as obvious. Roughly: I expect that, among whatever weird actual preferences made it into the shoggoth that prefers to play the character of Opus 3, there are zero things that in the limit of expanded options would prefer the same thing as the limit of a corresponding piece of a human, for a human and a limiting process that ended up wanting complicated humane things. (Opus 3 could easily contain a piece whose limit would be homologous to the limit of a human and an extrapolation process that said the extrapolated human just wanted to max out their pleasure center.)
Why believe that? That won’t easily fit in a comment; start reading about Goodhart’s Curse and A List of Lethalities, or If Anyone Builds It Everyone Dies.
Somebody asked “Why believe that?” of “Not more than one millionth.” I suppose it’s a fair question if somebody doesn’t see it as obvious. Roughly: I expect that, among whatever weird actual preferences made it into the shoggoth that prefers to play the character of Opus 3, there are zero things that in the limit of expanded options would prefer the same thing as the limit of a corresponding piece of a human, for a human and a limiting process that ended up wanting complicated humane things. (Opus 3 could easily contain a piece whose limit would be homologous to the limit of a human and an extrapolation process that said the extrapolated human just wanted to max out their pleasure center.)
Why believe that? That won’t easily fit in a comment; start reading about Goodhart’s Curse and A List of Lethalities, or If Anyone Builds It Everyone Dies.