The Chinchilla’s paper states that a 10T parameter model would require 1.30e+28 flops or 150 milion petaflop days. A state-of-the-art Nvdia DGX H100 requires 10 KW and it produces theoretically 8 petaflops FP16. With a training efficiency at 50% and a training time of 100 days, it would require 375,000 DGX H100 systems to train such model, for a total power required of 3.7 Gigawatt. That’s a factor of 100x larger any supercomputer in production today. Also, orchestrating 3 milion GPUs seems well beyond our engineering capabilities.
It seems unlikely we will see 10 T models trained like using the scaling law of the Chinchilla paper any time in the next 10 to 15 years.
The Chinchilla’s paper states that a 10T parameter model would require 1.30e+28 flops or 150 milion petaflop days. A state-of-the-art Nvdia DGX H100 requires 10 KW and it produces theoretically 8 petaflops FP16. With a training efficiency at 50% and a training time of 100 days, it would require 375,000 DGX H100 systems to train such model, for a total power required of 3.7 Gigawatt. That’s a factor of 100x larger any supercomputer in production today. Also, orchestrating 3 milion GPUs seems well beyond our engineering capabilities.
It seems unlikely we will see 10 T models trained like using the scaling law of the Chinchilla paper any time in the next 10 to 15 years.