I found this valuable, especially in the context of thinking about how smart humans actually are compared with other animals, and how this should impact our expectations for AI. I generally agree that we are deeply reliant on culture, in ways that aren’t obvious from the inside. I suspect that this is evidence in favour of slower takeoff speeds, because being as smart as humans isn’t nearly enough to do as well as humans. Any other thoughts in this direction?
Interestingly, I read this as evidence in favour of a fast takeoff, because being able to explicitly sort through the traditions for which ones were valuable and which ones were not, seems something a decent AI could do much better than a human. If part of our knowledge and power is held in forms we don’t understand, we become vulnerable to entities who do understand it.
Being able to speak is probably more important than being as smart as a human. Cultural / memetic evolution is orders of magnitude faster than biological, but its ability to function is dependent on having a memory better than mortal minds. Speech gives some limited non-mortal memory, as does writing the printing press, or the internet. These inventions enable more efficient evolution. AI will ramp up evolution to even higher speeds, since external memory will be replaced with internal 1) lossless and 2) intelligent memory. As such I am unconvinced that this would mean slower takeoff speeds. (You just explained that the most important factor in doing well as humans is something humans are not overly good at, instead of the special magic that only humans posess.)
I found this valuable, especially in the context of thinking about how smart humans actually are compared with other animals, and how this should impact our expectations for AI. I generally agree that we are deeply reliant on culture, in ways that aren’t obvious from the inside. I suspect that this is evidence in favour of slower takeoff speeds, because being as smart as humans isn’t nearly enough to do as well as humans. Any other thoughts in this direction?
Interestingly, I read this as evidence in favour of a fast takeoff, because being able to explicitly sort through the traditions for which ones were valuable and which ones were not, seems something a decent AI could do much better than a human. If part of our knowledge and power is held in forms we don’t understand, we become vulnerable to entities who do understand it.
Being able to speak is probably more important than being as smart as a human. Cultural / memetic evolution is orders of magnitude faster than biological, but its ability to function is dependent on having a memory better than mortal minds. Speech gives some limited non-mortal memory, as does writing the printing press, or the internet. These inventions enable more efficient evolution. AI will ramp up evolution to even higher speeds, since external memory will be replaced with internal 1) lossless and 2) intelligent memory. As such I am unconvinced that this would mean slower takeoff speeds. (You just explained that the most important factor in doing well as humans is something humans are not overly good at, instead of the special magic that only humans posess.)
I don’t see the connection between the latter claim and the former claim.