Not to put words in the author’s mouth, but when they said “We go gently...”, I don’t think they meant “go” as in become extinct, at least not any time soon. I took that to mean “go” into obscurity and stagnation instead of keeping on advancing technologically until we’re building Dyson spheres and colonizing other planets and all the science fiction stuff that most people believe humanity is going to do eventually. In that scenario, we would keep living on aimlessly for many millenia until some asteroid or other cosmic event took us out, because we had never advanced enough to be able to handle that or have colonies as a backup.
I agree with you that we’re unlikely to stop reproducing just because many humans get addicted to watching/interacting with content fed to us by a perfect algorithm for most of our waking hours. Raising a family seems to be one of those things that brings intrinsic meaning and pleasure to many people, so I believe you’d see more of it, not less—most of the reasons people are choosing not to have kids today are because they don’t have enough time or money in today’s economy and work environment, and in this scenario all those problems are solved. This scenario makes the assumption that the AI-fueled content machine would be so addictive that basically all humans would forsake all other pursuits and live like the people in WALL-E. I don’t think that’s necessarily true, and if it isn’t, we might see a population explosion requiring our AI-enabled oligarchic overlords to take control measures to keep it manageable.
Far from humanity going extinct, I think one possible catastrophe in the future, if AI advances roughly along these lines, is a Malthusian scenario where the population grows way beyond current levels thanks to AI optimizing the distribution of resources to make that possible, but becoming so dependent on complex AI logistics to provide everyone’s needs that any slight hiccup in the distribution network can quickly cause a famine that kills millions.
This scenario seems to allow enough room for AI alignment and humans still being in the driver’s seat on big picture issues that it wouldn’t decide to let us go extinct intentionally. We can hope.
Before you get too excited about the idea, let’s think for a minute. What would world leaders—notoriously a bunch of people prone to be ruthless, sociopathic, and morally unscrupulous, even if they’re ostensibly in charge of liberal democracies—be able to reach through their cultural boundaries and agree on?
Peace? No way. Everyone has too many problems like outstanding land disputes they want to reserve the option of using war to correct.
An end to poverty? For who? To any leader in the developed world, agreeing on human plenty and prosperity as supreme values that transcend national borders would involve giving up some of their resources to people in the third world who are clearly suffering more. Everyone’s resources are already stretched really thin with their existing projects as it is, so that’s a total non-starter.
Property rights? Maybe, as long as you didn’t get specific enough to make it mean anything. Any language that implied it was wrong for a government to take property from its citizens on any pretext it liked is certainly out.
Technological advancement for the betterment of humanity? Sure, but everyone’s doing that already. Even if all the world leaders got together and solemnly swore to focus their efforts on pushing the limits of science and disseminating their learning to the rest of the world, they would… keep doing exactly what they’re doing now, keeping secrets, only publishing what is convenient and making excuses about national security concerns every time they get called out about it.
So what could they agree on?
Law and order? Now we’re getting closer, but that is some problematic phrasing. What if this whole concept of “international law” gets applied to tell a world leader about how they can treat their own people? No one wants that. All our world leaders are quick to call each other out on their various human rights abuses, but we’ve all got skeletons (or repressed minority groups, as the case may be) in our own closet. So what’s the part of “law and order” that all world leaders could all agree on?
I honestly believe that if they had a summit like this, the outcome would be for all the leaders of the world to come together and formally agree that the supreme moral value of humanity is obedience and submission to the state. That’s the one thing that is in line with all of their desires, whether they want to admit it or not. The leaders of America and a few others with a freedom-loving image to keep up would have to make a show of complaint, but even they could rationalize it away.