This seems pretty well done! Some thoughts on future research in this direction:
It seems like you probably could have gotten certainty about compute for at least a handful of the models studied in question (either because the model was open sourced, or you have direct access to the org training it like Eleuther) -- it would be interesting to see how the estimation methods compared to the exact answer in this case. (Probably doable with GPT-J for example)
While I agree with dropout not significantly reducing computation I think two more contemporary techniques are worth considering here: structured sparsity in weights (‘blocksparse’), and mixture-of-experts gating (‘switch transformer’). I think the latter is more important because it changes both the training compute and inference compute.
Comparing custom ML hardware (e.g. Google’s TPUs or Baidu’s Kunlun, etc) is tricky to put on these sorts of comparisons. For those I think the MLPerf Benchmarks are super useful. I’d be curious to hear the authors’ expectations of how this research changes in the face of more custom ML hardware.
In general I think it’d be good to integrate a bunch of the performance benchmarks that are publicly available (since hardware providers are usually pretty eager to show off stats that make their hardware look good) into calibrations for this method. It’s also usually pretty straightforward to compute the operations and exact utilization in these runs, since they’re heavily standardized on the exact model and dataset.
It seems like you probably could have gotten certainty about compute for at least a handful of the models studied in question
We thought so too—but in practice it has been surprisingly hard. Profilers are surprisingly buggy. Our colleague Marious looked into this more in depth here.
Maybe we are just going the wrong way about it. If someone here figures out how to directly measure compute in eg a pytorch or TF model it would be a huge boon to us.
I think two more contemporary techniques are worth considering here: structured sparsity in weights (‘blocksparse’), and mixture-of-experts gating (‘switch transformer’)
Great suggestions! I think those would be a great future caveats to look into.
I’d be curious to hear the authors’ expectations of how this research changes in the face of more custom ML hardware.
My naive impression is that our conclusions do not change much. You would just need to plug into the effective performance (peak performance×utilization) in the second formula.
Probably the trickiest part might be figuring out the utilization rate for the custom hardware—though this is a general problem with the second method.
In general I think it’d be good to integrate a bunch of the performance benchmarks that are publicly available (since hardware providers are usually pretty eager to show off stats that make their hardware look good) into calibrations for this method.
I think that would be nice! We started a public spreadsheet with some info on different hardware. This might be of help to someone who wants to dig deeper into the topic!
Comparing custom ML hardware (e.g. Google’s TPUs or Baidu’s Kunlun, etc) is tricky to put on these sorts of comparisons. For those I think the MLPerf Benchmarks are super useful. I’d be curious to hear the authors’ expectations of how this research changes in the face of more custom ML hardware.
I’d be pretty excited to see more work on this. Jaime already shared our hardware sheet where we collect information on GPUs but as you outline that’s the peak performance and sometimes misleading.
This seems pretty well done! Some thoughts on future research in this direction:
It seems like you probably could have gotten certainty about compute for at least a handful of the models studied in question (either because the model was open sourced, or you have direct access to the org training it like Eleuther) -- it would be interesting to see how the estimation methods compared to the exact answer in this case. (Probably doable with GPT-J for example)
While I agree with dropout not significantly reducing computation I think two more contemporary techniques are worth considering here: structured sparsity in weights (‘blocksparse’), and mixture-of-experts gating (‘switch transformer’). I think the latter is more important because it changes both the training compute and inference compute.
Comparing custom ML hardware (e.g. Google’s TPUs or Baidu’s Kunlun, etc) is tricky to put on these sorts of comparisons. For those I think the MLPerf Benchmarks are super useful. I’d be curious to hear the authors’ expectations of how this research changes in the face of more custom ML hardware.
In general I think it’d be good to integrate a bunch of the performance benchmarks that are publicly available (since hardware providers are usually pretty eager to show off stats that make their hardware look good) into calibrations for this method. It’s also usually pretty straightforward to compute the operations and exact utilization in these runs, since they’re heavily standardized on the exact model and dataset.
Thank you Alex! You make some great points.
We thought so too—but in practice it has been surprisingly hard. Profilers are surprisingly buggy. Our colleague Marious looked into this more in depth here.
Maybe we are just going the wrong way about it. If someone here figures out how to directly measure compute in eg a pytorch or TF model it would be a huge boon to us.
Great suggestions! I think those would be a great future caveats to look into.
My naive impression is that our conclusions do not change much. You would just need to plug into the effective performance (peak performance×utilization) in the second formula.
Probably the trickiest part might be figuring out the utilization rate for the custom hardware—though this is a general problem with the second method.
I think that would be nice! We started a public spreadsheet with some info on different hardware. This might be of help to someone who wants to dig deeper into the topic!
I’d be pretty excited to see more work on this. Jaime already shared our hardware sheet where we collect information on GPUs but as you outline that’s the peak performance and sometimes misleading.
Indeed, the MLPerf benchmarks are useful. I’ve already gathered their data in this sheet and would love to see someone playing around with it. Next to MLPerf, Lambda Labs also shares some standardized benchmarks.