Let me clarify what I had in mind by “smart engineering”: sure, 8-high HBF is slower than 8-high HBM but you can supplement part of 12-high HBM with, say, 6-high HBF and get same or better bandwidth for a lower chip price but higher electricity consumption.[1]
As for the prefill, if you can allocate a certain share of hardware to prefill-only, you can minimize HBM on that hardware. However, there are usually ~3x less prefill than decode nodes and in practice even less than a quarter of hardware is “locked” into fixed pools for flexibility, so the economic effect will be limited (maybe that’s why research into heterogeneous hardware generally is so slow).
There’s also a latency aspect, but experts in the mid-to-late layers can be prefetched early in the forward pass, while those in the early layers can be predicted by the MTP heads on the previous token. Although both approaches come with some bandwidth penalty as the predictions will never be perfect
Let me clarify what I had in mind by “smart engineering”: sure, 8-high HBF is slower than 8-high HBM but you can supplement part of 12-high HBM with, say, 6-high HBF and get same or better bandwidth for a lower chip price but higher electricity consumption.[1]
As for the prefill, if you can allocate a certain share of hardware to prefill-only, you can minimize HBM on that hardware. However, there are usually ~3x less prefill than decode nodes and in practice even less than a quarter of hardware is “locked” into fixed pools for flexibility, so the economic effect will be limited (maybe that’s why research into heterogeneous hardware generally is so slow).
There’s also a latency aspect, but experts in the mid-to-late layers can be prefetched early in the forward pass, while those in the early layers can be predicted by the MTP heads on the previous token. Although both approaches come with some bandwidth penalty as the predictions will never be perfect