I think my point is that the breakthroughs can’t have a 5% chance of happening each year, this implies low-hanging fruit abounds in whatever the relevant bit of neuroscience is. If the brainlike AGI program has a 5% chance of becoming possible this year then it is viable this year. It can’t require that many incremental ideas, otherwise the relevant breakthrough would be much less likely this year.
Maybe I am confused about your point? Or we’re thinking about different counterfactuals?
OK, suppose we are 3 breakthroughs away from the brainlike AGI program and there’s a 15% chance of a breakthrough each year. I don’t think that changes the bottom line, which is that when the brainlike AGI program finally starts working, the speed at which it passes through the capabilities milestones is greater the later it starts working.
Now that’s just one paradigm of course, but I wonder if I could make a similar argument about many of the paradigms, and then argue that conditional on 2035 or 2045 timelines, AGI will probably be achieved via one of those paradigms, and thus takeoff will be faster.
(I suppose that brings up a whole nother intuition I should have mentioned, which is that the speed of takeoff probably depends on which paradigm is the relevant paradigm during the intelligence explosion, and that might have interesting correlations with timelines...)
I think my point is that the breakthroughs can’t have a 5% chance of happening each year, this implies low-hanging fruit abounds in whatever the relevant bit of neuroscience is. If the brainlike AGI program has a 5% chance of becoming possible this year then it is viable this year. It can’t require that many incremental ideas, otherwise the relevant breakthrough would be much less likely this year.
Maybe I am confused about your point? Or we’re thinking about different counterfactuals?
OK, suppose we are 3 breakthroughs away from the brainlike AGI program and there’s a 15% chance of a breakthrough each year. I don’t think that changes the bottom line, which is that when the brainlike AGI program finally starts working, the speed at which it passes through the capabilities milestones is greater the later it starts working.
Now that’s just one paradigm of course, but I wonder if I could make a similar argument about many of the paradigms, and then argue that conditional on 2035 or 2045 timelines, AGI will probably be achieved via one of those paradigms, and thus takeoff will be faster.
(I suppose that brings up a whole nother intuition I should have mentioned, which is that the speed of takeoff probably depends on which paradigm is the relevant paradigm during the intelligence explosion, and that might have interesting correlations with timelines...)