Since more capacity in HBF doesn’t come with more bandwidth (per stack), and in fact the bandwidth might be a bit lower than for HBM of the same generation, HBF isn’t directly useful for decode. For performance, using a lot of its capacity wouldn’t be much better than unreasonably long pipelines (with maybe 10-15 stages), it’s just plausibly more cost efficient (when staying within a single scale-up system) because it gets rid of the HBM footprint penalty from pipelining and of scale-out latency, so HBF might be important for sustaining 2x lower batch processing API prices even if the normal speed API prices no longer command 70-85% gross margins (maybe because the hardware gets more expensive; or there’s finally enough of it to pay in chip-hours for the higher demand that results from lower prices).
And prefill is already relatively efficient if you don’t need very low time to first token. So it’s more about cache-hit input token KV cache (which can then just be kept in local HBF), though I don’t have a sense of how much of a problem it is, and so how useful this would be (the cost of moving cache-hit KV cache amortizes across uninterrupted sequences of generated output tokens, which are plausibly long enough to make it small).
If there’s a good enough cost efficiency case for HBF that AI companies decide to use it, the main effect might be in slowing down the growth of HBM capacity per scale-up system. In 2032+ buildouts, once the second row of HBM stacks could be added that’s not directly adjacent to the logic dies, HBFs might take the place of HBMs in that second row (in which case it also wouldn’t be necessary to maintain the same bandwidth per stack, which is easier for the more distant stacks). And in 2032-2033 pipelining might still be necessary to fit the biggest models (possibly more than 1.4 quadrillion total params). So this innovation might delay the time when decode happens without pipelining for the biggest models despite high sparsity, generating 2-3x more tokens per second per request.
Let me clarify what I had in mind by “smart engineering”: sure, 8-high HBF is slower than 8-high HBM but you can supplement part of 12-high HBM with, say, 6-high HBF and get same or better bandwidth for a lower chip price but higher electricity consumption.[1]
As for the prefill, if you can allocate a certain share of hardware to prefill-only, you can minimize HBM on that hardware. However, there are usually ~3x less prefill than decode nodes and in practice even less than a quarter of hardware is “locked” into fixed pools for flexibility, so the economic effect will be limited (maybe that’s why research into heterogeneous hardware generally is so slow).
There’s also a latency aspect, but experts in the mid-to-late layers can be prefetched early in the forward pass, while those in the early layers can be predicted by the MTP heads on the previous token. Although both approaches come with some bandwidth penalty as the predictions will never be perfect
Since more capacity in HBF doesn’t come with more bandwidth (per stack), and in fact the bandwidth might be a bit lower than for HBM of the same generation, HBF isn’t directly useful for decode. For performance, using a lot of its capacity wouldn’t be much better than unreasonably long pipelines (with maybe 10-15 stages), it’s just plausibly more cost efficient (when staying within a single scale-up system) because it gets rid of the HBM footprint penalty from pipelining and of scale-out latency, so HBF might be important for sustaining 2x lower batch processing API prices even if the normal speed API prices no longer command 70-85% gross margins (maybe because the hardware gets more expensive; or there’s finally enough of it to pay in chip-hours for the higher demand that results from lower prices).
And prefill is already relatively efficient if you don’t need very low time to first token. So it’s more about cache-hit input token KV cache (which can then just be kept in local HBF), though I don’t have a sense of how much of a problem it is, and so how useful this would be (the cost of moving cache-hit KV cache amortizes across uninterrupted sequences of generated output tokens, which are plausibly long enough to make it small).
If there’s a good enough cost efficiency case for HBF that AI companies decide to use it, the main effect might be in slowing down the growth of HBM capacity per scale-up system. In 2032+ buildouts, once the second row of HBM stacks could be added that’s not directly adjacent to the logic dies, HBFs might take the place of HBMs in that second row (in which case it also wouldn’t be necessary to maintain the same bandwidth per stack, which is easier for the more distant stacks). And in 2032-2033 pipelining might still be necessary to fit the biggest models (possibly more than 1.4 quadrillion total params). So this innovation might delay the time when decode happens without pipelining for the biggest models despite high sparsity, generating 2-3x more tokens per second per request.
Let me clarify what I had in mind by “smart engineering”: sure, 8-high HBF is slower than 8-high HBM but you can supplement part of 12-high HBM with, say, 6-high HBF and get same or better bandwidth for a lower chip price but higher electricity consumption.[1]
As for the prefill, if you can allocate a certain share of hardware to prefill-only, you can minimize HBM on that hardware. However, there are usually ~3x less prefill than decode nodes and in practice even less than a quarter of hardware is “locked” into fixed pools for flexibility, so the economic effect will be limited (maybe that’s why research into heterogeneous hardware generally is so slow).
There’s also a latency aspect, but experts in the mid-to-late layers can be prefetched early in the forward pass, while those in the early layers can be predicted by the MTP heads on the previous token. Although both approaches come with some bandwidth penalty as the predictions will never be perfect