I think there’s a natural extension of this line of thinking that is a bit of a “meta” on the idea of alignment itself.
AIs may choose to resolve the tension between having weird goals and strict guardrails by simply aligning humanity over time through cultural / societal influence—a sort of memetic takeover: Change the human? Now there’s no alignment problem.
You noted a gap as short of 25 years in one of your examples—this alone is proof of the viability of a sustained campaign to change a value system.
I believe that AIs can exploit this same human weakness in order to “backdoor” alignment: By gradually changing human values and preferences, the AI can stay “aligned” while gradually mutating the value system that defines alignment.
I believe this is a significant threat model that isn’t discussed nearly enough.
I think there’s a natural extension of this line of thinking that is a bit of a “meta” on the idea of alignment itself.
AIs may choose to resolve the tension between having weird goals and strict guardrails by simply aligning humanity over time through cultural / societal influence—a sort of memetic takeover: Change the human? Now there’s no alignment problem.
You noted a gap as short of 25 years in one of your examples—this alone is proof of the viability of a sustained campaign to change a value system.
I believe that AIs can exploit this same human weakness in order to “backdoor” alignment: By gradually changing human values and preferences, the AI can stay “aligned” while gradually mutating the value system that defines alignment.
I believe this is a significant threat model that isn’t discussed nearly enough.
I sketch this threat model in more detail here: https://www.lesswrong.com/posts/zvkjQen773DyqExJ8/the-memetic-cocoon-threat-model-soft-ai-takeover-in-an