I didn’t really follow your argument for the relevance of this question to AI development. Why should raw innovation ability be more susceptible to discontinuous jumps than cultural processing ability? Until I understand the supposed relevance to AI better, it’s hard for me to say which of the two notions is more relevant for this purpose.
If our technological power comes from accumulated cultural power, then we can say that this power came from a slow, accumulated process over tens of thousands of years. This type of thing is much harder to unseat than if our power came from a simple architectural difference in ability to innovate.
The abilities we obtained from architectural changes to our brains also came from a slow, accumulated process, taking even longer than cultural evolution does.
Yes, but the actual power didn’t come from the architectural difference, even if the architectural difference got us the ability to process culture. The actual power came from the culture. That’s my point.
ETA: Are you saying that AI could discontinuously develop culture much more quickly than humans?
I guess what I was trying to say is (although I think I’ve partially figured out what you meant; see next paragraph), cultural evolution is a process that acquires adaptations slowly-ish and transmits previously-acquired adaptations to new organisms quickly, while biological evolution is a process that acquires adaptations very slowly and transmits previously-acquired adaptations to new organisms quickly. You seem to be comparing the rate at which cultural evolution acquires adaptations to the rate at which biological evolution transmits previously-acquired adaptations to new organisms, and concluding that cultural evolution is slower.
Re-reading the part of your post where you talked about AI takeoff speeds, you argue (which I hadn’t understood before) that the rise of humans was fast on an evolutionary timescale, and slow on a cultural timescale, so that if it was due to an evolutionary change, it must involve a small change that had a large effect on capabilities, so that a large change will occur very suddenly if we mimic evolution quickly, while if it was due to a cultural change, it was probably a large change, so mimicking culture quickly won’t produce a large effect on capabilities unless it is extremely quick.
This clarifies things, but I don’t agree with the claim. I think slow changes in the intelligence of a species is compatible with fast changes in its capabilities even if the changes are mainly in raw innovative ability rather than cultural learning. Innovations can increase ability to innovate, causing a positive feedback loop. A species could have high enough cultural learning ability for innovations to be transmitted over many generations without having the innovative ability to ever get the innovations that will kick off this loop. Then, when they start slowing gaining innovative ability, the innovations accumulated into cultural knowledge gradually increase, until they reach the feedback loop and the rate of innovation becomes more determined by changes in pre-existing innovations than by changes in raw innovative ability. There doesn’t even have to be any evolutionary changes in the period in which innovation rate starts to get dramatic.
If you don’t buy this story, then it’s not clear why the changes being in cultural learning ability rather than in raw innovative ability would remove the need for a discontinuity. After all, our cultural learning ability went from not giving us much advantage over other animals to “accumulating decisive technological dominance in an evolutionary eyeblink” in an evolutionary eyeblink (quotation marks added for ease of parsing). Does this mean our ability to learn from culture must have greatly increased from a small change? You argue in the post that there’s no clear candidate for what such a discontinuity in cultural learning ability could look like, but this seems just as true to me for raw innovative ability.
Perhaps you could argue that it doesn’t matter if there’s a sharp discontinuity in cultural learning ability because you can’t learn from a culture faster than the culture learns things to teach you. In this case, yes, perhaps I would say that AI-driven culture could make advancements that look discontinuous on a human scale. Though I’m not entirely sure what that would look like, and I admit it does sound kind of soft-takeoffy.
If our technological power comes from accumulated cultural power, then we can say that this power came from a slow, accumulated process over tens of thousands of years. This type of thing is much harder to unseat than if our power came from a simple architectural difference in ability to innovate.
The abilities we obtained from architectural changes to our brains also came from a slow, accumulated process, taking even longer than cultural evolution does.
Yes, but the actual power didn’t come from the architectural difference, even if the architectural difference got us the ability to process culture. The actual power came from the culture. That’s my point.
ETA: Are you saying that AI could discontinuously develop culture much more quickly than humans?
I guess what I was trying to say is (although I think I’ve partially figured out what you meant; see next paragraph), cultural evolution is a process that acquires adaptations slowly-ish and transmits previously-acquired adaptations to new organisms quickly, while biological evolution is a process that acquires adaptations very slowly and transmits previously-acquired adaptations to new organisms quickly. You seem to be comparing the rate at which cultural evolution acquires adaptations to the rate at which biological evolution transmits previously-acquired adaptations to new organisms, and concluding that cultural evolution is slower.
Re-reading the part of your post where you talked about AI takeoff speeds, you argue (which I hadn’t understood before) that the rise of humans was fast on an evolutionary timescale, and slow on a cultural timescale, so that if it was due to an evolutionary change, it must involve a small change that had a large effect on capabilities, so that a large change will occur very suddenly if we mimic evolution quickly, while if it was due to a cultural change, it was probably a large change, so mimicking culture quickly won’t produce a large effect on capabilities unless it is extremely quick.
This clarifies things, but I don’t agree with the claim. I think slow changes in the intelligence of a species is compatible with fast changes in its capabilities even if the changes are mainly in raw innovative ability rather than cultural learning. Innovations can increase ability to innovate, causing a positive feedback loop. A species could have high enough cultural learning ability for innovations to be transmitted over many generations without having the innovative ability to ever get the innovations that will kick off this loop. Then, when they start slowing gaining innovative ability, the innovations accumulated into cultural knowledge gradually increase, until they reach the feedback loop and the rate of innovation becomes more determined by changes in pre-existing innovations than by changes in raw innovative ability. There doesn’t even have to be any evolutionary changes in the period in which innovation rate starts to get dramatic.
If you don’t buy this story, then it’s not clear why the changes being in cultural learning ability rather than in raw innovative ability would remove the need for a discontinuity. After all, our cultural learning ability went from not giving us much advantage over other animals to “accumulating decisive technological dominance in an evolutionary eyeblink” in an evolutionary eyeblink (quotation marks added for ease of parsing). Does this mean our ability to learn from culture must have greatly increased from a small change? You argue in the post that there’s no clear candidate for what such a discontinuity in cultural learning ability could look like, but this seems just as true to me for raw innovative ability.
Perhaps you could argue that it doesn’t matter if there’s a sharp discontinuity in cultural learning ability because you can’t learn from a culture faster than the culture learns things to teach you. In this case, yes, perhaps I would say that AI-driven culture could make advancements that look discontinuous on a human scale. Though I’m not entirely sure what that would look like, and I admit it does sound kind of soft-takeoffy.