I agree that there’s something about human intelligence that’s more relevant than anything that’s been done before, but what specifically? Suppose for example that you don’t keep in mind the question of reflective stability. Then you’re probably quite prone to investigate things that will not stick around during self-modification and therefore are approximately irrelevant to the hard problem of alignment. I’m not exactly going to defend the old-school mathy research, because at least in retrospect one might have known it would be too hard / wouldn’t work, but it had its eye on the ball in the sense of understanding this sort of deep, general, difficult obstacle to understanding minds in the way we want to.
I agree that there’s something about human intelligence that’s more relevant than anything that’s been done before, but what specifically? Suppose for example that you don’t keep in mind the question of reflective stability. Then you’re probably quite prone to investigate things that will not stick around during self-modification and therefore are approximately irrelevant to the hard problem of alignment. I’m not exactly going to defend the old-school mathy research, because at least in retrospect one might have known it would be too hard / wouldn’t work, but it had its eye on the ball in the sense of understanding this sort of deep, general, difficult obstacle to understanding minds in the way we want to.