In The Secrets of Our Success, Joe Henrich argues that without our stock of cultural knowledge, individual humans are not particularly more generally intelligent than apes.
I 75% agree with this, but I do think that individual humans are smarter than individual chimpanzees. A big area of disagreement is distinguishing between “intrinsic ability to innovate” vs. “ability to process culture”, and whether it’s even possible to distinguish the two. I wrote a post about this two years ago.
For Henrich, the story of human uniqueness is a story of a feedback loop: increased cultural know-how, which drives genetic selection for bigger brains and better social learning, which leads to increased cultural know-how, which drives genetic selection for bigger brains….and so forth, until you have a very weird great ape that is weak, hairless, and has put a flag on the moon.
This is the big crux for me on the evolution of humans and its relevance to the foom debate.
Roughly, I think Henrich’s model is correct. I think his model provides a simple, coherent explanation for why humans dominate the world, and why it happened on such a short timescale, discontinuously with other animals.
Of course, intelligence plays a large role on his model: you can’t get ants who can go to the moon, no matter how powerful their culture. But the the great insight is that our power does not come from our raw intelligence: it comes from our technology/culture, which is so powerful because it was allowed to accumulate.
Cultural accumulation is a zero-to-one discontinuity. That is, you can go a long time without any of it, and then something comes along that’s able to do it just a little bit and then shortly after, it blows up. But after you’ve already reached one, going from “being able to accumulate culture at all” to “being able to accumulate it slightly faster” does not give you the same discontinuous foom as before.
We could, for example, imagine that an AI that can accumulate culture slightly faster than other humans. Since this AI is only slightly better than humans, however, it doesn’t go and create its own culture on its own. Unlike the humans—who actually did go and create their own culture completely on their own, separate from other animals—the AI will simply be one input to the human economy.
This AI would be important input to our economy for sure, but not a completely separate entity producing its own distinct civilization, like the prototypical AI that spins up nanobot factories and kills us all within 3 minutes. It will be more like the brilliant professor, or easily-copyable-worker. In other words, it might speed up our general civilizational abilities to develop technology, and greatly enhance our productive capabilities. But it won’t, on its own, discontinuously produce technology 2.0 (where 1.0 was humans and animals roughly are technology 0.0).
I think a superintelligent AI can FOOM its way to manufacturing nanobots because the biggest bottleneck to engineering and manufacturing those is research that can be done without needing input from the physical universe beyond the physics we already know, and the machines we already have, with very slight upgrades or creative usages beyond what they were designed for. Manufacturing nanobots is like a logic brain teaser for a sufficiently intelligent reasoner. I guess you have a different perspective in that you think the process requires a culture of socializing beings, and/or more input from the physical universe?
I 75% agree with this, but I do think that individual humans are smarter than individual chimpanzees. A big area of disagreement is distinguishing between “intrinsic ability to innovate” vs. “ability to process culture”, and whether it’s even possible to distinguish the two. I wrote a post about this two years ago.
This is the big crux for me on the evolution of humans and its relevance to the foom debate.
Roughly, I think Henrich’s model is correct. I think his model provides a simple, coherent explanation for why humans dominate the world, and why it happened on such a short timescale, discontinuously with other animals.
Of course, intelligence plays a large role on his model: you can’t get ants who can go to the moon, no matter how powerful their culture. But the the great insight is that our power does not come from our raw intelligence: it comes from our technology/culture, which is so powerful because it was allowed to accumulate.
Cultural accumulation is a zero-to-one discontinuity. That is, you can go a long time without any of it, and then something comes along that’s able to do it just a little bit and then shortly after, it blows up. But after you’ve already reached one, going from “being able to accumulate culture at all” to “being able to accumulate it slightly faster” does not give you the same discontinuous foom as before.
We could, for example, imagine that an AI that can accumulate culture slightly faster than other humans. Since this AI is only slightly better than humans, however, it doesn’t go and create its own culture on its own. Unlike the humans—who actually did go and create their own culture completely on their own, separate from other animals—the AI will simply be one input to the human economy.
This AI would be important input to our economy for sure, but not a completely separate entity producing its own distinct civilization, like the prototypical AI that spins up nanobot factories and kills us all within 3 minutes. It will be more like the brilliant professor, or easily-copyable-worker. In other words, it might speed up our general civilizational abilities to develop technology, and greatly enhance our productive capabilities. But it won’t, on its own, discontinuously produce technology 2.0 (where 1.0 was humans and animals roughly are technology 0.0).
I think a superintelligent AI can FOOM its way to manufacturing nanobots because the biggest bottleneck to engineering and manufacturing those is research that can be done without needing input from the physical universe beyond the physics we already know, and the machines we already have, with very slight upgrades or creative usages beyond what they were designed for. Manufacturing nanobots is like a logic brain teaser for a sufficiently intelligent reasoner. I guess you have a different perspective in that you think the process requires a culture of socializing beings, and/or more input from the physical universe?