It is weird to me that so many people who have thought hard about AI don’t think that human emulations are a better bet for a good future than LLMs, if we had that choice. Human emulations have many features that make me a lot more hopeful that they would preserve value in the universe and also not get everyone killed, and it seems obvious that they both have and would be afforded moral value. I do agree that there is a large probability that the emulation scenario goes sideways, and Hanson’s Age of Em is not an optimistic way for that to play out, but we don’t have to let things play out that way. With Ems we would definitely at least have a fighting chance.
The basic reason for this is basically that starting from a human doesn’t actually buy you that much in terms of alignment, because the reason alignment is such a nasty problem is mostly preserved from AIs to EMs.
The 2 big issues are this:
Humans are broadly misaligned to each other, and there’s both a technical and political nightmare if you would want to align EMs to a level that doesn’t imply most biological humans are dead, and while control can work, it’s not likely that the political will to control EMs will exist, unfortunately.
@Vladimir_Nesov and @dr_s explains why misalignment is a bigger deal post-AGI than now, and that’s due to the fact that once you can overwrite a person’s desires/not depend on other beings anymore, the instrumentally convergent action stops being beneficial:
More generally, the idea that humans/WBEs are safer than AIs when scaled up in power as much as AIs are rest on very questionable assumptions at best.
The basic reason for this is basically that starting from a human doesn’t actually buy you that much in terms of alignment, because the reason alignment is such a nasty problem is mostly preserved from AIs to EMs.
The 2 big issues are this:
Humans are broadly misaligned to each other, and there’s both a technical and political nightmare if you would want to align EMs to a level that doesn’t imply most biological humans are dead, and while control can work, it’s not likely that the political will to control EMs will exist, unfortunately.
@Vladimir_Nesov and @dr_s explains why misalignment is a bigger deal post-AGI than now, and that’s due to the fact that once you can overwrite a person’s desires/not depend on other beings anymore, the instrumentally convergent action stops being beneficial:
https://www.lesswrong.com/posts/Z8C29oMAmYjhk2CNN/non-superintelligent-paperclip-maximizers-are-normal#FTfvrr9E6QKYGtMRT
https://www.lesswrong.com/posts/2ujT9renJwdrcBqcE/the-benevolence-of-the-butcher
@RogerDearnaley has a great post on the issue of uploads being misaligned:
https://www.lesswrong.com/posts/4gGGu2ePkDzgcZ7pf/3-uploading#Humans_are_Not_Aligned
More generally, the idea that humans/WBEs are safer than AIs when scaled up in power as much as AIs are rest on very questionable assumptions at best.