A better framing for my understanding is “it will appear as if we solved alignment”, so all humans will be persuaded to upload and the few scientists/programmers who discover that uploads are running in “optimized” mode as a single forward pass without the promised internal computation “functionally equivalent to consciousness” will have been disappeared 5 years ago...
The benefit of the gradual disempowerment frame is about what to look for (persuasion, trust in marketing materials without anyone “wanting” independent reviews, natural selection of ever-more-anti-bio-human ideologies, …) while the framing of “solving alignment” sounds to me like an impossible task with infinitely large attack surface area. But the old-school AI safety doomerisim is not mutually exclusive, just non-practical ontology (which leads to “only math is real, physics is boring, mentalistic talk is emergent, collective behaviour is well explained by game theory, VNM rationality ought to be morally real, proofs are possible (in principle)” kind of thinking that doesn’t translate to actionable recommendations).
A better framing for my understanding is “it will appear as if we solved alignment”, so all humans will be persuaded to upload and the few scientists/programmers who discover that uploads are running in “optimized” mode as a single forward pass without the promised internal computation “functionally equivalent to consciousness” will have been disappeared 5 years ago...
The benefit of the gradual disempowerment frame is about what to look for (persuasion, trust in marketing materials without anyone “wanting” independent reviews, natural selection of ever-more-anti-bio-human ideologies, …) while the framing of “solving alignment” sounds to me like an impossible task with infinitely large attack surface area. But the old-school AI safety doomerisim is not mutually exclusive, just non-practical ontology (which leads to “only math is real, physics is boring, mentalistic talk is emergent, collective behaviour is well explained by game theory, VNM rationality ought to be morally real, proofs are possible (in principle)” kind of thinking that doesn’t translate to actionable recommendations).