Doesn’t the context length matter a lot, here? Like, in the tables you gave above, none of the agents seem to be hitting the limits of the context window, and thus triggering compaction, whereas in the 130 billion output tokens for Navier-Stokes, a single agent would’ve been dominated by its compaction mechanism. Thus, even setting aside the speedup, it seems plausible to me that the lambda parameter changes quite a lot based on task size. On a very hard problem requiring billions of output tokens, it seems plausible that swarms have an edge. (Or not! I don’t know. My point is that I think extrapolating from ~100k output token cases might not work.)
Yes, I think this does matter a lot. Extremely long CoTs (beyond 1 million tokens) will degrade in some manner due to compaction, which will make the graph of score vs tokens spent become shallower (and could be what forces it to plateau after millions of tokens). This will lessen the gap between the two and may eventually make swarm scaling exceed CoT scaling.
This appears in my analysis where I say:
There may also be other advantages to multiple agents on top of speed. For example, if you keep increasing the length of the chain of thought for a single agent, the performance eventually plateaus. But the height of the plateau for a 1,000-agent swarm may be greater than for a single agent.
But ideally I would have mentioned compaction as a specific mechanism for this.
yes. gains from swarm use are time-based without a pooled context (which we don’t know how to do effectively). not that time-based improvements are anything to sneer at, but they aren’t improvements in core reasoning.
Doesn’t the context length matter a lot, here? Like, in the tables you gave above, none of the agents seem to be hitting the limits of the context window, and thus triggering compaction, whereas in the 130 billion output tokens for Navier-Stokes, a single agent would’ve been dominated by its compaction mechanism. Thus, even setting aside the speedup, it seems plausible to me that the lambda parameter changes quite a lot based on task size. On a very hard problem requiring billions of output tokens, it seems plausible that swarms have an edge. (Or not! I don’t know. My point is that I think extrapolating from ~100k output token cases might not work.)
Yes, I think this does matter a lot. Extremely long CoTs (beyond 1 million tokens) will degrade in some manner due to compaction, which will make the graph of score vs tokens spent become shallower (and could be what forces it to plateau after millions of tokens). This will lessen the gap between the two and may eventually make swarm scaling exceed CoT scaling.
This appears in my analysis where I say:
But ideally I would have mentioned compaction as a specific mechanism for this.
yes. gains from swarm use are time-based without a pooled context (which we don’t know how to do effectively). not that time-based improvements are anything to sneer at, but they aren’t improvements in core reasoning.