We might describe the result as “going out with a whimper.” Human reasoning gradually stops being able to compete with sophisticated, systematized manipulation and deception which is continuously improving by trial and error; human control over levers of power gradually becomes less and less effective; we ultimately lose any real ability to influence our society’s trajectory. By the time we spread through the stars our current values are just one of many forces in the world, not even a particularly strong one.
Man, in this scenario it really matters how much “our” AI systems are suffering or having enjoyable-on-their-terms experiences.
Man, in this scenario it really matters how much “our” AI systems are suffering or having enjoyable-on-their-terms experiences.