I think you probably need to understand many things about minds a lot better than “evolution + genetics” understands biology before it makes much sense to try attacking questions about alignment mechanics in particular. To stick with the analogy, I suspect you might at least need the sort of mastery level where you understand Mitochondria and DNA transcription well enough to build your own basic functional versions of them from scratch before you can even really get started.
I agree that ‘we are confused about agency’ is not a good slogan for pointing to this inadequacy. I think ‘we haven’t advanced practical mind science to anywhere near the level we’ve advanced e.g. condensed matter physics’ is true and a blocker for alignment of superintelligence, but ‘we are confused about agency’ brings up much stronger associations around memes like ‘maybe Bayesian EV maximisation is conceptually wrong even in the idealised setting’ to me. These meme groups seem sufficiently distinct to merit separate slogans.
I think you probably need to understand many things about minds a lot better than “evolution + genetics” understands biology before it makes much sense to try attacking questions about alignment mechanics in particular. To stick with the analogy, I suspect you might at least need the sort of mastery level where you understand Mitochondria and DNA transcription well enough to build your own basic functional versions of them from scratch before you can even really get started.
I agree that ‘we are confused about agency’ is not a good slogan for pointing to this inadequacy. I think ‘we haven’t advanced practical mind science to anywhere near the level we’ve advanced e.g. condensed matter physics’ is true and a blocker for alignment of superintelligence, but ‘we are confused about agency’ brings up much stronger associations around memes like ‘maybe Bayesian EV maximisation is conceptually wrong even in the idealised setting’ to me. These meme groups seem sufficiently distinct to merit separate slogans.