5: Try to build aligned (or controlled) autonomous ASI agents
Unfortunately 5 seems to be profitable, in the short term, so I guess that’s what we’re doing. In fact, the “plan” seems to be the exact reverse or so of my preference ordering on what it should be.
AI danger is not about AI, it’s about governance. A sane civilization would be able to robustly defer and then navigate AI danger when it’s ready. AI is destabilizing, and while aligned AI (in a broad sense) is potentially a building block for a competent/aligned civilization (including human civilization), that’s only if it’s shaped/deployed in a competent/aligned way. Uploads are destabilizing in a way similar to AI (since they can be copied and scaled), even though they by construction ensure some baseline of alignment.
Intelligence amplification for biological humans (that can’t be copied) seems like the only straightforward concrete plan that’s not inherently destabilizing. But without highly speculative too-fast methods it needs AI danger to be deferred for a very long time, with a ban/pause that achieves escape velocity (getting stronger rather than weaker over time, for example by heavily restricting semi manufacturing capabilities). This way, there is hope for a civilization that eventually gets sufficiently competent to navigate AI danger, but the premise of a civilization sufficiently competent to defer AI danger indefinitely is damning.
I feel like intelligence amplification is plenty destabilising. Consider how toxic intelligence discourse is or has been right now already:
some people argue that some ethnic groups (usually their own) has inherently higher intelligence, which makes them better
other people who want to push back against the former then go the other extreme and claim intelligence doesn’t exist at all as an even partially measurable quantity
And what would you do with your intelligence amplification method? Sell it? So now richer people, and richer countries, are the ones to first reap the benefits, amplifying gaps in inequality which again have destabilising effects.
A lot of this ends up in similar places as aligned ASI, if you only consider the political side of it. Similar issues.
Similar issues but less extreme. The degree of concentration of power, misalignment risk, etc. from intelligence amplification would be smaller than from AGI.
It would be slower for sure, at least, being bound to human dynamics. But “same problems but slower” isn’t the same as a solution/alternative. Admittedly better in the limited sense that it’s less likely to end with straight up extinction, but it’s a rather grim world either way.
Since it’s slower, the tech development cycle is faster in comparison. Tech development --> less expensive tech --> more access --> less concetration of power --> more moral outcomes.
Mind uploading is probably quite hard. See here. It’s probably much easier to get AGI from partial understanding of how to do uploads, than to get actual uploads. Even if you have unlimited political capital, such that you can successfully prevent making partial-upload-AGIs, it’s probably just very technically difficult. Intelligence amplification is much more doable because we can copy a bunch of nature’s work by looking at all the existing genetic variants and their associated phenotypes.
We can certainly do intelligence amplification on the way to mind uploading. I would prefer to perform uploading before moving on to building (other) ASI, rather than ONLY intelligence amplification.
We almost certainly want to eventually do uploading, if nothing else because that’s probably how you avoid involuntary-preheatdeath-death. It might be the best way to do supra-genomic HIA, but I would rather leave that up to the next generation, because it seems both morally fraught and technically difficult. It’s far from clear to me that we ever want to make ASI; why ever do that rather than just have more human/humane personal growth and descendants? (I agree with the urgency of all the mundane horrible stuff that’s always happening; but my guess is we can get out of that stuff with HIA before it’s safe to make ASI. Alignment is harder than curing world hunger and stopping all war, probably (glib genie jokes aside).)
I think it’s just a matter of what’s more technologically achievable. Building LLMs turned out to be a lot easier than understanding neuroscience to a level even remotely close to what’s necessary to achieve 1 or 2. And both of those also require huge political capital due to needing (likely dangerous) human experimentation that would currently be considered unacceptable.
My preference ordering over approaches to surviving the next decade, assuming I had unlimited political capital, is (don’t build ASI and):
1: Solve mind uploading
2: … or at least intelligence amplification
(before proceeding, and then maybe do)
3: Imitation learning (with radically improved theoretical foundations)
4: … or corrigibility / oracles
(and don’t)
5: Try to build aligned (or controlled) autonomous ASI agents
Unfortunately 5 seems to be profitable, in the short term, so I guess that’s what we’re doing. In fact, the “plan” seems to be the exact reverse or so of my preference ordering on what it should be.
AI danger is not about AI, it’s about governance. A sane civilization would be able to robustly defer and then navigate AI danger when it’s ready. AI is destabilizing, and while aligned AI (in a broad sense) is potentially a building block for a competent/aligned civilization (including human civilization), that’s only if it’s shaped/deployed in a competent/aligned way. Uploads are destabilizing in a way similar to AI (since they can be copied and scaled), even though they by construction ensure some baseline of alignment.
Intelligence amplification for biological humans (that can’t be copied) seems like the only straightforward concrete plan that’s not inherently destabilizing. But without highly speculative too-fast methods it needs AI danger to be deferred for a very long time, with a ban/pause that achieves escape velocity (getting stronger rather than weaker over time, for example by heavily restricting semi manufacturing capabilities). This way, there is hope for a civilization that eventually gets sufficiently competent to navigate AI danger, but the premise of a civilization sufficiently competent to defer AI danger indefinitely is damning.
I feel like intelligence amplification is plenty destabilising. Consider how toxic intelligence discourse is or has been right now already:
some people argue that some ethnic groups (usually their own) has inherently higher intelligence, which makes them better
other people who want to push back against the former then go the other extreme and claim intelligence doesn’t exist at all as an even partially measurable quantity
And what would you do with your intelligence amplification method? Sell it? So now richer people, and richer countries, are the ones to first reap the benefits, amplifying gaps in inequality which again have destabilising effects.
A lot of this ends up in similar places as aligned ASI, if you only consider the political side of it. Similar issues.
Similar issues but less extreme. The degree of concentration of power, misalignment risk, etc. from intelligence amplification would be smaller than from AGI.
It would be slower for sure, at least, being bound to human dynamics. But “same problems but slower” isn’t the same as a solution/alternative. Admittedly better in the limited sense that it’s less likely to end with straight up extinction, but it’s a rather grim world either way.
Since it’s slower, the tech development cycle is faster in comparison. Tech development --> less expensive tech --> more access --> less concetration of power --> more moral outcomes.
Mind uploading is probably quite hard. See here. It’s probably much easier to get AGI from partial understanding of how to do uploads, than to get actual uploads. Even if you have unlimited political capital, such that you can successfully prevent making partial-upload-AGIs, it’s probably just very technically difficult. Intelligence amplification is much more doable because we can copy a bunch of nature’s work by looking at all the existing genetic variants and their associated phenotypes.
We can certainly do intelligence amplification on the way to mind uploading. I would prefer to perform uploading before moving on to building (other) ASI, rather than ONLY intelligence amplification.
We almost certainly want to eventually do uploading, if nothing else because that’s probably how you avoid involuntary-preheatdeath-death. It might be the best way to do supra-genomic HIA, but I would rather leave that up to the next generation, because it seems both morally fraught and technically difficult. It’s far from clear to me that we ever want to make ASI; why ever do that rather than just have more human/humane personal growth and descendants? (I agree with the urgency of all the mundane horrible stuff that’s always happening; but my guess is we can get out of that stuff with HIA before it’s safe to make ASI. Alignment is harder than curing world hunger and stopping all war, probably (glib genie jokes aside).)
Is this due to it basically being a Pareto frontier of political capital needed vs probability of causing doom, or some other reason?
I think it’s just a matter of what’s more technologically achievable. Building LLMs turned out to be a lot easier than understanding neuroscience to a level even remotely close to what’s necessary to achieve 1 or 2. And both of those also require huge political capital due to needing (likely dangerous) human experimentation that would currently be considered unacceptable.