Something I see a lot in high-end ML is papers that are very simple conceptually, but very tricky to get working properly. I’d imagine that having the dozen or so guys who know all the hyperparameter tweaks to get public algorithm X to run accounts for a lot of ‘secret sauce’. Lots of people were wondering how DALL-E worked, but diffusion for quality image synthesis had been around for a while at the time, for instance.
Also, off the top of my head, I can’t think of an instance where a lab had an entire secret algorithm locked up that was the basis for their lead. Feels like it’s always been public papers getting used to their full potential.
I guess like all things we will know for sure once the open chinese labs start doing it.
Something I see a lot in high-end ML is papers that are very simple conceptually, but very tricky to get working properly. I’d imagine that having the dozen or so guys who know all the hyperparameter tweaks to get public algorithm X to run accounts for a lot of ‘secret sauce’. Lots of people were wondering how DALL-E worked, but diffusion for quality image synthesis had been around for a while at the time, for instance.
Also, off the top of my head, I can’t think of an instance where a lab had an entire secret algorithm locked up that was the basis for their lead. Feels like it’s always been public papers getting used to their full potential.
What a timeline, eh?