Well, yeah, those things are all true. But at the same time, I mean, c’mon! Look at how Ev left things 100,000 years ago, and then look at the lives we live today. It’s a pretty wild change! I think it’s fair to say that modern humans are doing things, and wanting things, that Ev would never have expected.
Yeah, but there are also a lot of regularities that are preserved across the distributional shift. Humans are still avoiding pain and injury, for instance, which was both highly prioritized in the original distribution, and still highly prioritized after the invention and scale up of civilization, even though there are totally different threats that modern humans need to contend with (eg car accidents).
I think that if you did this well on the alignment of an AGI, the AGIs after learning a ton and massively scaling up their capabilities would be doing some wild and unexpected stuff, but they wouldn’t go around intentionally driving humans extinct to use their atoms for something else. “Don’t kill humans” is gonna be extremely highly prioritized in the reinforcement of almost all AI systems.
Yeah, but there are also a lot of regularities that are preserved across the distributional shift. Humans are still avoiding pain and injury, for instance, which was both highly prioritized in the original distribution, and still highly prioritized after the invention and scale up of civilization, even though there are totally different threats that modern humans need to contend with (eg car accidents).
I think that if you did this well on the alignment of an AGI, the AGIs after learning a ton and massively scaling up their capabilities would be doing some wild and unexpected stuff, but they wouldn’t go around intentionally driving humans extinct to use their atoms for something else. “Don’t kill humans” is gonna be extremely highly prioritized in the reinforcement of almost all AI systems.