GPU datacenters are different and closer to one giant machine, but that’s a recent phenomenon with centralized training of AI.
It’s worth distinguishing big datacenters (especially for pretraining, where scale-out networks need to be unusually good) from big scale-up systems/pods. Big scale-up systems (currently mostly rack-level, though TPUs were multi-rack for many years now) are motivated by very big numbers of total params in MoE models, not by centralized training of AI. Centralized training of AI (in the sense of pretraining) instead motivates big datacenters with good scale-out networks, but the individual scale-up systems within these datacenters could be small (even for models with a lot of total params). So these are completely different desiderata, calling for technologically unrelated things.
A mainframe is a single large very powerful machine. A datacenter is a bunch of servers (the PC model) that are conveniently colocated.
That’s a good point, regarding the CPU datacenters (though I don’t see how the analogy could transfer something useful at this level, if the conclusion isn’t already accepted and the analogy just illustrates what it looks like based on a more familiar story).
It’s worth distinguishing big datacenters (especially for pretraining, where scale-out networks need to be unusually good) from big scale-up systems/pods. Big scale-up systems (currently mostly rack-level, though TPUs were multi-rack for many years now) are motivated by very big numbers of total params in MoE models, not by centralized training of AI. Centralized training of AI (in the sense of pretraining) instead motivates big datacenters with good scale-out networks, but the individual scale-up systems within these datacenters could be small (even for models with a lot of total params). So these are completely different desiderata, calling for technologically unrelated things.
That’s a good point, regarding the CPU datacenters (though I don’t see how the analogy could transfer something useful at this level, if the conclusion isn’t already accepted and the analogy just illustrates what it looks like based on a more familiar story).