Most of this seems to be subsumed in the general question of how do you do research, and there’s lot of advice, but it’s (ironically) not at all a science. From my limited understanding of what goes on in the research groups inside these companies, it’s a combination of research intuition, small scale testing, checking with others and discussing the new approach, validating your ideas, and getting buy-in from people higher up that it’s worth your and their time to try the new idea. Which is the same as research generally.
At that point, I’ll speculate and assume whatever idea they have is validated in smaller but still relatively large settings. For things like sample efficiency, they might, say, train a GPT-3 size model, which now cost only a fraction of the researcher’s salary to do. (Yes, I’m sure they all have very large compute budgets for their research.) If the results are still impressive, I’m sure there is lots more discussion and testing before actually using the method in training the next round of frontier models that cost huge amounts of money—and those decisions are ultimately made by the teams building those models, and management.
Most of this seems to be subsumed in the general question of how do you do research, and there’s lot of advice, but it’s (ironically) not at all a science. From my limited understanding of what goes on in the research groups inside these companies, it’s a combination of research intuition, small scale testing, checking with others and discussing the new approach, validating your ideas, and getting buy-in from people higher up that it’s worth your and their time to try the new idea. Which is the same as research generally.
At that point, I’ll speculate and assume whatever idea they have is validated in smaller but still relatively large settings. For things like sample efficiency, they might, say, train a GPT-3 size model, which now cost only a fraction of the researcher’s salary to do. (Yes, I’m sure they all have very large compute budgets for their research.) If the results are still impressive, I’m sure there is lots more discussion and testing before actually using the method in training the next round of frontier models that cost huge amounts of money—and those decisions are ultimately made by the teams building those models, and management.
Thanks for this answer! Interesting. It sounds like the process may be less systematized than how I imagined it to be.