Agreed. There’s really a continuum here, where the rate of self-improvement can range from sublinear to fully exponential. Processes that lack a mechanism for feeding improvements back into the system’s underlying intelligence are unlikely to achieve exponential growth; they’ll tend to plateau or progress linearly at best.
One takeaway is that an effective regulatory regime aimed at limiting fast-takeoff risk might focus precisely on this distinction — restricting processes that iteratively improve at greater than a linear rate.
Agreed. There’s really a continuum here, where the rate of self-improvement can range from sublinear to fully exponential. Processes that lack a mechanism for feeding improvements back into the system’s underlying intelligence are unlikely to achieve exponential growth; they’ll tend to plateau or progress linearly at best.
One takeaway is that an effective regulatory regime aimed at limiting fast-takeoff risk might focus precisely on this distinction — restricting processes that iteratively improve at greater than a linear rate.