I doubt it’s all that qualitatively different than the sorts of summits humanity has surmounted before
This seems to imply that we have surmounted the fields of physics, that all available knowledge in all subfields has been acquired whereas the most that can be claimed is that we have reduced the degree of our ignorance in some of those subfields. We have not—by any stretch of the imagination—mastered the field. Indeed, if we think we cannot push our understanding of AI, and related alignment problems, further than our current degree of understanding of physics, I think that is a strong point for the “stop everything, while we still can” case.
Maybe a naive question: is it actually realistic to expect the possibility of slow take-off speeds? Once a computer becomes an AGI doesn’t it almost immediately, given its computing power, become an ASI? Again: not a technical person—but I am trying to imagine the average human (IQ: 100), the brain of which would be suddenly be connected to the computing power of a super computer. Wouldn’t he very shortly become an ASI?