10. AI systems will ultimately be wildly superhuman, and there probably won’t be strong technological hurdles right around human level. Extrapolating the rate of existing AI progress suggests you don’t get too much time between weak AI systems and very strong AI systems, and AI contributions could very easily go from being a tiny minority of intellectual work to a large majority over a few years.
I think there will be substantial technical hurdles along the lines of getting in-principle highly capable AI systems to reliably do what we want them to, these will probably be “commercially relevant” (i.e. not just a concern for X-risk researchers), and it’s plausible (though far from certain) that hurdles of this type will slow the rate of AI progress.
One reason I think this might slow the rate of progress is that important parts of this can’t be delegated to highly in-principle capable systems. One reason I think it might not slow progress much is that I don’t currently see a good reason why most of this work couldn’t be delegated to highly actually-capable systems.
I think there will be substantial technical hurdles along the lines of getting in-principle highly capable AI systems to reliably do what we want them to, these will probably be “commercially relevant” (i.e. not just a concern for X-risk researchers), and it’s plausible (though far from certain) that hurdles of this type will slow the rate of AI progress.
One reason I think this might slow the rate of progress is that important parts of this can’t be delegated to highly in-principle capable systems. One reason I think it might not slow progress much is that I don’t currently see a good reason why most of this work couldn’t be delegated to highly actually-capable systems.