I don’t think there’s need for an AGI to build a (separate) successor per se. Humans need the technological AGI only due to inability to copy/evolve our minds in a more efficient way compared to the existing biological one
I think that sort of ‘copying’ process counts as building a successor. More broadly, there’s a class of problems that center around “how can you tell whether changes to your thinking process make you better or worse at thinking?”, which I think you can model as imagining replacing yourself with two successors, one of which makes that change and the other of which doesn’t. [Your imagination can only go so far, tho, as you don’t know how those thoughts will go without actually thinking them!]
I meant ‘copying’ above only necessary in the human case to escape the slow evolving biological brain. While it is certainly available to a hypothetical AGI, it is not strictly necessary for self-improvement (at least copying of the whole AGI isn’t)
I don’t think there’s need for an AGI to build a (separate) successor per se. Humans need the technological AGI only due to inability to copy/evolve our minds in a more efficient way compared to the existing biological one
I think that sort of ‘copying’ process counts as building a successor. More broadly, there’s a class of problems that center around “how can you tell whether changes to your thinking process make you better or worse at thinking?”, which I think you can model as imagining replacing yourself with two successors, one of which makes that change and the other of which doesn’t. [Your imagination can only go so far, tho, as you don’t know how those thoughts will go without actually thinking them!]
I meant ‘copying’ above only necessary in the human case to escape the slow evolving biological brain. While it is certainly available to a hypothetical AGI, it is not strictly necessary for self-improvement (at least copying of the whole AGI isn’t)