A good term for 10^20 FLOP would be useful. This would make modern models around 100k to 10 million of this unit, which is a tangible number. Some people, e.g. at DeepMind tried to make “petaflop-days” (8.64e19) a thing but it didn’t catch on.
The 100k to 10M range is populated by abstract quantities—I think that for a measure to be useful here, it has to be imaginable.
Avogadro’s number has the benefit of historical precedent for describing quantities, and the coincidental property of allowing us to represent present-day training runs with numbers we see in the real world (outside of screens or print) when used as a denominator. It too might cease to be useful once exponents become necessary to describe training runs in terms of mol FLOPs.
Tired of making sense of exponents? Introducing: the mol FLOP!
Simply divide the size of a training run by Avogadro’s constant. Some examples:
AlexNet (2012): 2 µmol FLOPs
GPT-3 (2020): 0.5 mol FLOPs
Grok 4 (2025): 400 mol FLOPs
Bonus: The ballpark equivalent water volume for each, mapping 1 FLOP to 1 water molecule,
AlexNet (2012): 36 nL (tiny droplet)
GPT-3 (2020): 9 mL (two teaspoons)
Grok 4 (2025): 7.2 L (water cooler jug)
A good term for 10^20 FLOP would be useful. This would make modern models around 100k to 10 million of this unit, which is a tangible number. Some people, e.g. at DeepMind tried to make “petaflop-days” (8.64e19) a thing but it didn’t catch on.
H100 hours (or H100-equivalent hours) caught up to some extent and are imo a good unit (imo even better than mol FLOPs or petaflop days)
The 100k to 10M range is populated by abstract quantities—I think that for a measure to be useful here, it has to be imaginable.
Avogadro’s number has the benefit of historical precedent for describing quantities, and the coincidental property of allowing us to represent present-day training runs with numbers we see in the real world (outside of screens or print) when used as a denominator. It too might cease to be useful once exponents become necessary to describe training runs in terms of mol FLOPs.