P(DOOM from AGI | No Action) = 0. We will not die by AGI if we take no action to develop AGI, at least in less than astronomical timescales.
This is obviously not what will happen. Many groups will rush to develop AGI in the fastest and most powerful manner that they can possibly arrange, with safety largely treated as a public relations exercise and not a serious issue. My personal feeling is that any AGI developed in the next 30 years will very likely (p > 0.9) be sufficiently misaligned that humanity would be extinguished if it were scaled up to superintelligence.
My hopes for our species rest on one of three foundations: that I’m hopelessly miscalibrated about how hard alignment is, that AGI takes so much longer to develop that alignment is adequately solved, or that scaling to superintelligence is for some reason very much harder than developing AGI.
The second seems less likely each year, and the third seems an awfully flimsy hope. Both algorithm advances and hardware capabilities seem to offer substantial improvements well into the superhuman realm for all of the many narrow tasks that machines can now do better than humans, and I don’t see any reason to expect that they cannot scale at least as well for more general ones once AGI is achieved.
I do have some other outlier possibilities for hope, but they seem more based in fiction than reality.
So that mainly leaves the first hope, that alignment is actually a lot simpler and easier than it appears to me. It is at least worth checking.
P(DOOM from AGI | No Action) = 0. We will not die by AGI if we take no action to develop AGI, at least in less than astronomical timescales.
This is obviously not what will happen. Many groups will rush to develop AGI in the fastest and most powerful manner that they can possibly arrange, with safety largely treated as a public relations exercise and not a serious issue. My personal feeling is that any AGI developed in the next 30 years will very likely (p > 0.9) be sufficiently misaligned that humanity would be extinguished if it were scaled up to superintelligence.
My hopes for our species rest on one of three foundations: that I’m hopelessly miscalibrated about how hard alignment is, that AGI takes so much longer to develop that alignment is adequately solved, or that scaling to superintelligence is for some reason very much harder than developing AGI.
The second seems less likely each year, and the third seems an awfully flimsy hope. Both algorithm advances and hardware capabilities seem to offer substantial improvements well into the superhuman realm for all of the many narrow tasks that machines can now do better than humans, and I don’t see any reason to expect that they cannot scale at least as well for more general ones once AGI is achieved.
I do have some other outlier possibilities for hope, but they seem more based in fiction than reality.
So that mainly leaves the first hope, that alignment is actually a lot simpler and easier than it appears to me. It is at least worth checking.