I think that it’s one of the ways in which the world could be better prepared to build ASI in a few decades, but there are other ways too (e.g. burden of proof shifting, e.g. public epistemics improving, e.g. open-source lie detectors or privacy-preserving monitoring being widespread)
I find it a bit strange to list these things alongside HIE, because they seem obviously much weaker to me. (With the possible exception of “public epistemics improving”, that could be super important, but also not going to happen in a general/powerful form without widespread adoption of HIE?) As an intuition pump: if the global IQ distribution was shifted 3 SD lower, we wouldn’t be anywhere close to being able to have a conversation like this at all; I’d guess such a world would never be aware of the dangers of ASI even if it somehow managed to eventually build it. On the flipside, a world where the IQ distribution was shifted 3 SD higher would probably have all sorts of options that we aren’t even aware of.
there are a bunch of ways the world could be worse-prepared to build ASI even if there’s some HIE (e.g. what if the superbabies just race as fast as possible to ASI, and go significantly faster because they are smarter? What if the neuralink enhancements also tend to drive people slightly insane, and the scientists are working on safer versions but in the meantime a million people are using the unsafe versions and racing to build ASI in their basements?)
I don’t see why anywhere near the majority of enhanced people would want to race for ASI, even assuming we can’t also substantially enhance Wisdom? But I agree some of the enhanced people could end up wanting to do this, and that could be an issue, though the majority of enhanced people would probably want to stop them. But I think we’d be better off in a world with +3 SD IQ, by the reversal test. Also there are probably all sorts of governance things you could do to decrease these risks. This is a discussion we’d actually need to put some effort into having.
I’m guessing the underlying disagreement is that I think alignment is way harder than you do? As far as I know we could be further away from understanding how to do alignment than the Ancient Greeks were from being able to land on the moon.[1]
Maybe we shouldn’t be aiming to “build it in a few decades” at all? Maybe the enhanced people can find ways of delaying ASI for longer? E.g. maybe instead of de novo ASI we upload some smart and wise people to be emulated at high speed, and trust them to self modify into a entity that will prevent the creation of ASI, while we scale up the uploading tech for anyone else that wants it. But mostly, I think we should defer these questions to smarter and wiser future people, except for whatever informs the decisions we actually need to make now.
it’s a topic that’ll raise a lot of hackles and so I’d only want to do it if I actually thought it was important
My understanding is that you recommended A over S in part because you don’t see what we’d do differently with the extra time? And I’m like, HIE is what we’d do differently with the extra time. There might be simple, common sense ways of making the point that most people could appreciate: e.g. you wouldn’t want to make an important and irreversible life decision while drunk or tired.
Thanks for the detailed response!
I find it a bit strange to list these things alongside HIE, because they seem obviously much weaker to me. (With the possible exception of “public epistemics improving”, that could be super important, but also not going to happen in a general/powerful form without widespread adoption of HIE?) As an intuition pump: if the global IQ distribution was shifted 3 SD lower, we wouldn’t be anywhere close to being able to have a conversation like this at all; I’d guess such a world would never be aware of the dangers of ASI even if it somehow managed to eventually build it. On the flipside, a world where the IQ distribution was shifted 3 SD higher would probably have all sorts of options that we aren’t even aware of.
I don’t see why anywhere near the majority of enhanced people would want to race for ASI, even assuming we can’t also substantially enhance Wisdom? But I agree some of the enhanced people could end up wanting to do this, and that could be an issue, though the majority of enhanced people would probably want to stop them. But I think we’d be better off in a world with +3 SD IQ, by the reversal test. Also there are probably all sorts of governance things you could do to decrease these risks. This is a discussion we’d actually need to put some effort into having.
I’m guessing the underlying disagreement is that I think alignment is way harder than you do? As far as I know we could be further away from understanding how to do alignment than the Ancient Greeks were from being able to land on the moon.[1]
Maybe we shouldn’t be aiming to “build it in a few decades” at all? Maybe the enhanced people can find ways of delaying ASI for longer? E.g. maybe instead of de novo ASI we upload some smart and wise people to be emulated at high speed, and trust them to self modify into a entity that will prevent the creation of ASI, while we scale up the uploading tech for anyone else that wants it. But mostly, I think we should defer these questions to smarter and wiser future people, except for whatever informs the decisions we actually need to make now.
My understanding is that you recommended A over S in part because you don’t see what we’d do differently with the extra time? And I’m like, HIE is what we’d do differently with the extra time. There might be simple, common sense ways of making the point that most people could appreciate: e.g. you wouldn’t want to make an important and irreversible life decision while drunk or tired.
H/T Tsvi BT for this analogy