It still seems to me that recursive self improvement should be qualitatively different
Could you say more about this intuition?
When I think of an AGI system engaging in RSI, I imagine it reasoning in a very general, first principles sort of way that, “I want to improve myself, and I run on TPUs, so I should figure out how to make better TPUs.” But at the point that we have a system that can do that, we’re already going to be optimizing the fuck out of TPUs. And we’re going to be using all our not-quite-so-general systems to search for other things to optimize.
So where does the discontinuity (or curve shape change) come from?
Could you say more about this intuition?
When I think of an AGI system engaging in RSI, I imagine it reasoning in a very general, first principles sort of way that, “I want to improve myself, and I run on TPUs, so I should figure out how to make better TPUs.” But at the point that we have a system that can do that, we’re already going to be optimizing the fuck out of TPUs. And we’re going to be using all our not-quite-so-general systems to search for other things to optimize.
So where does the discontinuity (or curve shape change) come from?