Curious that neither Opus 5 nor Sol 5.6 spontaneously understands the pipeline parallelism arguments in the third paragraph[1] (the training data isn’t preparing them for prosaic RSI yet; Fable 5 is somewhat better). They think a pipeline reads everything, so that 4-stage 13 ms HBM means 52 ms. This is incorrect, since inactive KV cache of requests currently at the other stages isn’t read, it takes less time than 13 ms per rack to read weights plus requests active at that stage. So just 33 ms for a 4-stage pipeline (4 parts out of 8 in weights, plus 1 part out of the 8 in active KV cache), ignoring between-racks latency (which might make 3 rather than 4 racks of Rubin Oberon more closely match a single GB300 Oberon rack).
And if half of HBM is weights, while only 1 in 4 requests are active, then a request needs to pay for 4x as much weight reading as it pays for KV cache reading. So the cost of decode for a 40 TB model is 2.5x higher with 4-stage pipelining using 20 TB scale-up systems than it is with a single 80 TB scale-up system, and to partially compensate you want fewer weights. Only with 2.5x smaller models (16 TB, taking up 20% of total HBM rather than half) do the requests pay as much for reading weights as they do for reading KV cache. This is still 1.6x more weights than without pipelining (with the same 20 TB scale-up system), but far from 4x. The difference is more about what’s possible if you don’t care about cost too much, so that you can actually get to 2-3x more weights than what fits in half of one scale-up system (at 1.2-1.7x the cost compared to a twice-as-big-as-the-model single scale-up system), while without pipelining you are stuck below 1.6x (at 2.5x the cost) if you fill up to 80% of HBM capacity with weights.
Asking the models helps to both get a sense of whether my arguments are “objectively understandable” in some sense, and of the extent to which the models are still sometimes failing when their breadth of background knowledge should in principle allow them to succeed.
Curious that neither Opus 5 nor Sol 5.6 spontaneously understands the pipeline parallelism arguments in the third paragraph [1] (the training data isn’t preparing them for prosaic RSI yet; Fable 5 is somewhat better). They think a pipeline reads everything, so that 4-stage 13 ms HBM means 52 ms. This is incorrect, since inactive KV cache of requests currently at the other stages isn’t read, it takes less time than 13 ms per rack to read weights plus requests active at that stage. So just 33 ms for a 4-stage pipeline (4 parts out of 8 in weights, plus 1 part out of the 8 in active KV cache), ignoring between-racks latency (which might make 3 rather than 4 racks of Rubin Oberon more closely match a single GB300 Oberon rack).
And if half of HBM is weights, while only 1 in 4 requests are active, then a request needs to pay for 4x as much weight reading as it pays for KV cache reading. So the cost of decode for a 40 TB model is 2.5x higher with 4-stage pipelining using 20 TB scale-up systems than it is with a single 80 TB scale-up system, and to partially compensate you want fewer weights. Only with 2.5x smaller models (16 TB, taking up 20% of total HBM rather than half) do the requests pay as much for reading weights as they do for reading KV cache. This is still 1.6x more weights than without pipelining (with the same 20 TB scale-up system), but far from 4x. The difference is more about what’s possible if you don’t care about cost too much, so that you can actually get to 2-3x more weights than what fits in half of one scale-up system (at 1.2-1.7x the cost compared to a twice-as-big-as-the-model single scale-up system), while without pipelining you are stuck below 1.6x (at 2.5x the cost) if you fill up to 80% of HBM capacity with weights.
Asking the models helps to both get a sense of whether my arguments are “objectively understandable” in some sense, and of the extent to which the models are still sometimes failing when their breadth of background knowledge should in principle allow them to succeed.