Meta, with 350k H100s, assuming sxm models with 989 terra flops rated, training in dense fp32 (so half perf) assuming they dedicate half the cluster to a single model and train for a year, and have 2⁄3 utilization, would have put 10^27 flops into the model by eoy.
Assuming Nvidia is correct that the next generation is 5 times faster, a B100 cluster purchased over 2025 and 2026 at the same investment rate as today (approximately 10-20 billion annually) would hit 10^28 flops by EOY 2027.
I am also going to note that Nvidia stands to earn well over 100 billion annually with the B100 now they are bundling it with hardware. They have a lot of incentive to lobby for no compute limit, and the funds to pay the lobbyists.
Presumably AMD/Intel/other USA IC companies have the same directional incentive.
“Columbus, Ohio, into what CEO Pat Gelsinger described to reporters on Tuesday as “the largest AI chip manufacturing site in the world”, starting as soon as 2027.”
Compute limits likely prevent models from being useful enough to justify these investments.
Does the case for x-risk stand up to the arguments these wealthy and successful us companies are going to predictably make? Such as “how do you know n flops is enough for the model to be dangerous. Show me a model on your computer that is too dangerous to control”.
Meta, with 350k H100s, assuming sxm models with 989 terra flops rated, training in dense fp32 (so half perf) assuming they dedicate half the cluster to a single model and train for a year, and have 2⁄3 utilization, would have put 10^27 flops into the model by eoy.
Assuming Nvidia is correct that the next generation is 5 times faster, a B100 cluster purchased over 2025 and 2026 at the same investment rate as today (approximately 10-20 billion annually) would hit 10^28 flops by EOY 2027.
I am also going to note that Nvidia stands to earn well over 100 billion annually with the B100 now they are bundling it with hardware. They have a lot of incentive to lobby for no compute limit, and the funds to pay the lobbyists.
Presumably AMD/Intel/other USA IC companies have the same directional incentive.
Update: https://www.reuters.com/technology/intel-prepares-100-bln-spending-spree-across-four-us-states-2024-03-20/
“Columbus, Ohio, into what CEO Pat Gelsinger described to reporters on Tuesday as “the largest AI chip manufacturing site in the world”, starting as soon as 2027.”
Compute limits likely prevent models from being useful enough to justify these investments.
Does the case for x-risk stand up to the arguments these wealthy and successful us companies are going to predictably make? Such as “how do you know n flops is enough for the model to be dangerous. Show me a model on your computer that is too dangerous to control”.