agreed that among all paths to good things that I see, a common thread is somehow uplifting human cognition to keep pace with advanced AI. however, I doubt that that’s even close to good enough—human cooperation is shaky and unreliable. most humans who think they’d do good things if made superintelligent probably are wrong due to various ways to value drift when the structure of one’s cognition changes, and many humans who say they think they’d do good things are simply lying, rather than deluding themselves or overestimating their own durable-goodness. it seems to me that in order to make this happen, we need to make AIs that strongly want all humans and humanity and etc emergent groups to stick around, the way a language model wants to output text.
Yes, wanting humans to stick around is probably a minimal necessity, but it’s still probably not enough—as the post explains.
And I simply don’t think you can “make humans smarter” in ways that don’t require, or at least clearly risk, erasing things that make us fundamentally human as we understand it today.
agreed that among all paths to good things that I see, a common thread is somehow uplifting human cognition to keep pace with advanced AI. however, I doubt that that’s even close to good enough—human cooperation is shaky and unreliable. most humans who think they’d do good things if made superintelligent probably are wrong due to various ways to value drift when the structure of one’s cognition changes, and many humans who say they think they’d do good things are simply lying, rather than deluding themselves or overestimating their own durable-goodness. it seems to me that in order to make this happen, we need to make AIs that strongly want all humans and humanity and etc emergent groups to stick around, the way a language model wants to output text.
Yes, wanting humans to stick around is probably a minimal necessity, but it’s still probably not enough—as the post explains.
And I simply don’t think you can “make humans smarter” in ways that don’t require, or at least clearly risk, erasing things that make us fundamentally human as we understand it today.