Were you implying that Chinchilla represents algorithmic progress? If so, I disagree: technically you could call a scaling law function an algorithm, but in practice, it seems Chinchilla was better because they scaled up the data. There are more aspects to scale than model size.
I think it’s distinct from something like Tree of thought. We have ideas that are trivial but enabled by greater compute vs novel ideas that would have worked at earlier levels of compute.
Fair enough. But for the purposes of this post, the point is that capability increased without increased compute. If you prefer, bucket it as “compute” vs “non-compute” instead of “compute” vs “algorithmic”.
Were you implying that Chinchilla represents algorithmic progress? If so, I disagree: technically you could call a scaling law function an algorithm, but in practice, it seems Chinchilla was better because they scaled up the data. There are more aspects to scale than model size.
Scaling up the data wasn’t algorithmic progress. Knowing that they needed to scale up the data was algorithmic progress.
It seems particularly trivial from an algorithmic aspect? You have the compute to try an idea so you try it. The key factor is still the compute.
Unless you’re including the software engineering efforts required to get these methods to work at scale, but I doubt that?
I think whether or not it’s trivial isn’t the point: they did it, it worked, and they didn’t need to increase the compute to make it happen.
I think it’s distinct from something like Tree of thought. We have ideas that are trivial but enabled by greater compute vs novel ideas that would have worked at earlier levels of compute.
Fair enough. But for the purposes of this post, the point is that capability increased without increased compute. If you prefer, bucket it as “compute” vs “non-compute” instead of “compute” vs “algorithmic”.