I think Jeremy is pointing to something real, but I think “interpolation versus extrapolation” in the strict mathematical sense is not quite the correct pointer to what he’s observing.
For another thing, consider that billions of humans over thousands of years invented language, math, science, technology, and everything in the $100T economy, 100% autonomously and from scratch, with no new training data dropped from the heavens. Is that “good extrapolation”? Not really. I don’t think there’s anything in the 30000 BC “training data” that would be “extrapolated” into the ability to travel to the moon and build advanced microchips and so on. I think the metaphor should be more like “creation” or “building”, not “extrapolation”.
I think Jeremy is pointing to something real, but I think “interpolation versus extrapolation” in the strict mathematical sense is not quite the correct pointer to what he’s observing.
For one thing, in high-dimensional space, everything is extrapolation.
For another thing, consider that billions of humans over thousands of years invented language, math, science, technology, and everything in the $100T economy, 100% autonomously and from scratch, with no new training data dropped from the heavens. Is that “good extrapolation”? Not really. I don’t think there’s anything in the 30000 BC “training data” that would be “extrapolated” into the ability to travel to the moon and build advanced microchips and so on. I think the metaphor should be more like “creation” or “building”, not “extrapolation”.