There’s reason to expect λ can be substantially increased, and maybe even brought above 1,[1] by the right orchestration and for some tasks, even keeping in mind the good caveats you give about ex. the Navier-Stokes swarm.
“Scaling Discovery through Test-Time Communication”announced today finds that while swarms aren’t too useful on ex. Terminal-Bench, even Sonnet 4.6 swarms can outperform independent agents with equivalent compute budget, specifically when the task is verifiable and requires some creativity (ex. ARC-AGI-3, FrontierCS Polyomino Packing, MNIST Compression).
Further caveats:
They only tested up to 5 agents, so not exactly an OpenAI-style 10k swarm, and it’s likely there are diminishing returns past some N of agents.
The ARC-AGI-3 result is less exciting than it looks because it’s counting that benchmark’s “actions”, not the tokens to perform them.
Otherwise Fig. 4 would actually demonstrate a case where λ > 1!
Below a certain minimum compute budget (necessary for communication overhead?) swarms do worse.
The paper’s own model caps λ at 1.
It’s possible that swarm advantages decrease with stronger models, though per Kim et al. this might be because stronger models can just solve the tasks outright and thus benefits less from sharing intermediate results. Today’s paper by Park et al. specifically chose to pick a weaker model to run ARC-AGI-3 on.
In other words, swarms aren’t useful if a single agent can just one-shot the task. Sort of how like you’re better off putting one programmer on a task they can complete in an afternoon, than putting 5 on it.
Un-caveats:
Some tasks could only be completed by communicating agents.
Agents were all the same in each test, so very homogeneous, but still managed to make and share useful individual breakthroughs via a bit of clever protocol work.
Re: λ = 0.5 (like AI Futures Model) being “very much in line with estimates from economists for the diminishing returns of human teams”, I think this is the wrong anchor.
On priors, human teams can have λ > 1 in some cases; that is, there are tasks where having four people work the problem for a week is better than having one person work the problem for a month.[2] The advantage comes from pursuing different directions and sharing intermediate results/creative ideas, based on individual experience/skills. I think this sort of thing is why Anthropic is now assigning persistent identities to their 30,000 internal agents.
I suspect the main thing dragging λ down historically is agent homogeneity, but agents also have potential advantages in coordination that humans don’t, like better trust mechanisms, lower transaction costs, skill/memory sharing, and deliberate usage of decision theory.
Finally, to do handwavy analysis of your backed-out λ values:
BrowseComp: = 0.68, 90% CI = [0.63, 0.76] SEC-Bench Pro: = 0.57, 90% CI = [0.52, 0.61] Terminal-Bench: = 0.48, 90% CI = [0.40, 0.57]
It’s notable that Terminal-Bench showed no swarm benefit in the paper above, and that’s also the lowest λ you found. The other OpenAI 16-agent cases were also using Ultra mode, which by my understanding is more of a traditional hierarchical multi-agent system than the freely-communicating systems that Anthropic and OpenAI are moving towards. So we might expect this to translate into:
Tasks not suited for swarms (ex. very serial): λ ≈ 0.5 Tasks where hierarchical multi-agent swarms are useful: λ ≈ 0.6 or 0.7 Tasks where communicating swarms can share meaningful intermediate progress: λ ≈ 1 After some future advancements in agent coordination (where task is not trivial for single agent): λ > 1
Maybe a naive question, I’m having trouble understanding how λ > 1 could be possible in principle; if a 10 agent swarm can do 11 times the work, then could I not improve my single agents by serializing this parallelization? (I think I’m just parroting some bog-standard argument in parallel computing.) That is, have my single meta-agent act as agent 1 for a while, then agent 2 for a while, and so on, the same way a single-core CPU simulates multi-threading.
Initially I thought that a swarm might win out against the above setup in efficiency by saving on the cost of context switching, but I would guess that loading up a new agent has the at least the same (and possibly higher) cost as context switching an existing one?
No that’s a great question, thanks! The paper I discussed above actually agrees with you, that’s why their model of how this works caps λ at 1.
My counterargument is that this is true for an idealized serial agent. We can refer to the human example again for intuition:[1]
There are tasks where having 4 people work the problem for 10 hours each in a week is better than having 1 person work the problem for 40 hours in that week.
We can see that λ > 1 could be true in a couple core scenarios:
The 4 people have different skillsets that complement each other, while the 1 person has just one skillset.
All 5 people have the same skillset, but they take different approaches due to random chance/coordination/initial conditions/experience, and one of those approaches is more fruitful. Assuming a large number of possible approaches, it’s more likely the team finds a better one.
With optimal approach switching, the 1 person could attempt four different approaches in the same amount of time, making it even odds, but that isn’t really how people work in practice. Idealized vs. real.
You can also see here why agent homogeneity is so problematic.
Re: efficiency on context switching, I think you’re right that swarm has no efficiency there, optimally. But of course one of agents’ main flaws (and humans’ main flaws?) is getting too attached to the current context/approach/belief.[2]
Reformulated to use version from footnote of previous comment, which in retrospect I prefer to avoid confusion about task speedup via parallelization, which is a separate matter.
Though you could just employ a setup that wipes the serial agent’s context entirely, and denies it access to previous work. Luckily we can’t do that to humans!
I see, I agree that it’s possible that a particular AI design easily gets caught in its context and periodically wiping it like an anterograde amnesiac might actually make it do better.
I guess it feels like if we ever had a model that was λ > 1 that would signal some serious inadequacy of the model that feels not hard to RL out?
A good theoretical question! If you have a set of humans with λ > 1, is that a sign of serious inadequacies in some or all of them, which ought to be addressed? Possibly!… if you want to optimize for task success, anyway.[1]
Thinking more about our context-switching serial agent:
On tasks where sharing intermediate results is very valuable (ex. hard research tasks), you really want the “trained to message other agents about intermediate results, and know how to trust/accept/reject those messages” behavior Anthropic/OpenAI are currently pursuing. Theoretically you could implement that by just having the serial agent send messages to its future selves. But the RL to teach that is multi-agent system RL! And the easiest implementation is real-life swarms.[2]
To put that another way, I expect we’ll have agents trained to send messages in a multi-agent system in an effective way. If you run a practical test with just one of those agents that has no one to message, against the multi-agent system, then you can get λ > 1. There will be a theoretical setup in which a serial agent matches the multi-agent system performance, but it won’t be used in practice.[3]
Thinking more about heterogeneity:
There’s an interesting path where λ > 1 in variants of a single model. A possible future: we have some basic continual learning using overnight weights-updating for agents working different jobs/roles, a more advanced version of Anthropic’s current long-running agent identities setup. There is no single agent that knows everything and that could substitute for all of them, because getting all of that context into the weights crowds model identity/skills/context/J-space in some way that makes them less effective.[4]
That all said, you’ve clarified my thinking here and I’ve updated down on λ > 1 thanks to this conversation.
In the limit, this almost feels like talking about whether a single grand unified consciousness is superior to a civilization, or an ASI singleton vs. ASI swarms, though we would have to carefully decompose “superior” there.
Though you could implement this in your serial agent by giving it the ability to end its current session with a “wait for reply/new message” tool. There might be some tricky caching issues with that but perhaps surmountable.
What does this mean for forecasting? I think the practical effects here mean that the practical speedup can still be equivalent to λ > 1. That is, parallelism can deliver increasing marginal returns (again, at least within some bounds on N agents) on forecasted available compute compared to the old pre-swarm baseline. However, you could just adjust the baseline to the ideal multi-agent-system-simulating serial agent, which clamps λ ≤ 1.
But it’s plausible no such limit exists in practice, or that more heterogeneity than this is needed, aka you need substantially different training runs to really get near-human variety.
Interesting. My initial intuition was that an agent which randomizes u.a.r. over all permutations of approaches should perform at least as well as the swarm. But this isn’t true! If we have 4 approaches, 3 of which lead to an hour of wasted time and 1 which leads to instant success, then a swarm of 4 agents will always be instantly successful. While a single agent will waste 0, 1, 2 or 3 hours, for an average of 1.5h wasted.
This is only theoretical of course. The example is chosen to be very extreme. Also, in real life we have communication overhead, the need for a “main” agent to verify the results, etc. But I think it’s a good intuition pump.
with randomized algos like SAT solvers, you can simulate early stopping of parallel runs by a round robin on a single thread, so in principle and in the limit you shouldn’t be able to get superlinear scaling just from theory.. sure, in practice, on actual hardware and software, there will be sweet spots a la L1 cache physical distance from the logic core, or the whole reason GPUs exist … but I wouldn’t count on the sweet spot that will exist for ~10 agents to then scale to ~10 million agents with a failure of competitive pressure to find improvements to single/few-agent performance in the next month
I think an ideal serialized agent wouldn’t be more constrained by working memory than a swarm would be. It’s not handling all the context all at once, it’s context-switching. It can emulate the full swarm (and even messaging behavior!) that way, see more discussion of that here.
But then again, we saw that kind of logic break down in practice in multi-threading on CPUs because of caching. In theory, a single-core CPU can emulate an N-core CPU in N× as much time. In practice, constantly swapping out the whole CPU L1/L2 cache wastes time, so a single-core CPU takes longer. Or rather, an N-core CPU is more than N× better; λ > 1.
So this comes back to the discussion on context-switching costs. The simple case is that you switch once per swarm agent the serial agent is simulating, and in that case caching costs can be equal. But the organized swarm case requires switching most every time the serialized agent wants to send or receive a message, which would be a lot more cache-swapping than running the agents in parallel… and with a lot bigger caches than CPUs have. So ex. if you have a nearly-full context window of 900k tokens, and you send a message after every 5k tokens of reasoning before swapping out the whole cache, you’re in trouble![1]
However even then, I suspect this might be less relevant than heterogeneity.
Agents are only now starting to be trained to cooperate with each other and self-organize.
In my experience today’s agents frequently make obvious coordination mistakes and fail to use good theory of mind or have high-trust patterns of communication when prompting each other. They feel like they are optimized for doing everything themselves and at best delegating very specific tasks to subagents.
I haven’t looked into SEC-Bench Pro, but if you take a benchmark that can be done pretty well by a single agent, it makes sense that it’s not that parallelizable, but things like “running a large corporation” clearly can’t be done without some mechanism to divide attention. For LLMs, things work somewhat differently, since you can clone agents and share context more cheaply, but it is definitely not going to make sense to chuck all the inputs of your company into a single smart agent’s context and have them make every decision.
Also, time does matter. If an agent could have solved Navier Stokes by carefully maintaining a large file system for itself and working on each subproblem one at a time, maybe it could produce the whole Lean proof in ~5 years. There are many kinds of problems where that would not be a success.
Civilization basically just is effective mechanisms of dividing labor so that a bunch of people can focus their finite attention/context windows on doing one thing well at a time. It would not work if you only had one guy trying to do everything at once, even if that guy were really competent.
I think humans kind of suck at cooperating with each other and self-governing, compared to what is possible. The difference between the least effective teams of humans and the most effective teams, comprised of all the same people with different communication norms, already spans orders of magnitude, but even the most effective humans are more limited in trust and communication bandwidth than LLMs could/likely soon will be.
Thanks Julian, that’s very thoughtful. I’ve now read your post on this and I thought it was good.
I’m definitely less bullish than you are about getting λ>1, though it may be possible, especially in cases where agent diversity is important (and where that can be implemented somehow) or where it is hard for a single agent to try out lots of different approaches successively without them contaminating each other.
You may be interested to know that Phil Trammell is thinking about something similar to increasing λ in the essay below. In fact he sees understanding parallelisability as key to understanding the prospects on an intelligence explosion:
Ah, a much more formal take that decomposes the relevant factors, I gather. I’ll look into that, thanks!
Edit: also FYI I haven’t looked deeply yet, but Opus 5.5 System Card would seem to still show λ in the 0.5 to 0.7 range, like I imagine you would predict.
Self-Organizing Agent Teams (SAT) outperform their strongest member, compute-matched inference by the strongest individual agent, and perfect routing over independent answers.
This sounds like λ > 1, but I think it’s more an exercise in caution on skimming arXiv preprints, as I don’t think this is good evidence:
Their linearization test is executing the strategies developed by the multi-agent, multi-model team of 3. Not apples to apples. A multi-agent, single-model team of 3 does worse* (though frustratingly it’s also executing given strategies). Performance gains may partly come from heterogeneity.[1]
Though Table 4 tries to demonstrate they come at least partly from the self-organized team structures.
*Except for one benchmark (BIG-Bench Extra Hard) where it does better for some reason.
Also “does better/worse” is quite possibly mostly inside confidence intervals here… which aren’t reported. (!)
“swarm” sizes here are just 3, and models are old (ex. Sonnet 4). Sample size also kind of small.
Compute-matching is loose; no token counts given, word “approximately” is used.
Headline result applies only to some of the benchmarks they tested.
On the whole I’m not comfortable updating much on this evidence. (also in terms of self-organizing teams results, unfortunately these folks were scooped by a more interesting and relevant result from Anthropic’s Opus 5.5 System Card, Ctrl-F for “Emergent team structures”)
You know I’ve found it useful myself to compare/contrast multiple LLMs’ results on programming tasks. I wonder if there might be gains to fine-tuning a lot of variants of a single frontier model, or if you really need different pretrains to get value.
There’s reason to expect λ can be substantially increased, and maybe even brought above 1,[1] by the right orchestration and for some tasks, even keeping in mind the good caveats you give about ex. the Navier-Stokes swarm.
“Scaling Discovery through Test-Time Communication” announced today finds that while swarms aren’t too useful on ex. Terminal-Bench, even Sonnet 4.6 swarms can outperform independent agents with equivalent compute budget, specifically when the task is verifiable and requires some creativity (ex. ARC-AGI-3, FrontierCS Polyomino Packing, MNIST Compression).
Further caveats:
They only tested up to 5 agents, so not exactly an OpenAI-style 10k swarm, and it’s likely there are diminishing returns past some N of agents.
The ARC-AGI-3 result is less exciting than it looks because it’s counting that benchmark’s “actions”, not the tokens to perform them.
Otherwise Fig. 4 would actually demonstrate a case where λ > 1!
Below a certain minimum compute budget (necessary for communication overhead?) swarms do worse.
The paper’s own model caps λ at 1.
It’s possible that swarm advantages decrease with stronger models, though per Kim et al. this might be because stronger models can just solve the tasks outright and thus benefits less from sharing intermediate results. Today’s paper by Park et al. specifically chose to pick a weaker model to run ARC-AGI-3 on.
In other words, swarms aren’t useful if a single agent can just one-shot the task. Sort of how like you’re better off putting one programmer on a task they can complete in an afternoon, than putting 5 on it.
Un-caveats:
Some tasks could only be completed by communicating agents.
Agents were all the same in each test, so very homogeneous, but still managed to make and share useful individual breakthroughs via a bit of clever protocol work.
Re: λ = 0.5 (like AI Futures Model) being “very much in line with estimates from economists for the diminishing returns of human teams”, I think this is the wrong anchor.
On priors, human teams can have λ > 1 in some cases; that is, there are tasks where having four people work the problem for a week is better than having one person work the problem for a month.[2] The advantage comes from pursuing different directions and sharing intermediate results/creative ideas, based on individual experience/skills. I think this sort of thing is why Anthropic is now assigning persistent identities to their 30,000 internal agents.
I suspect the main thing dragging λ down historically is agent homogeneity, but agents also have potential advantages in coordination that humans don’t, like better trust mechanisms, lower transaction costs, skill/memory sharing, and deliberate usage of decision theory.
Finally, to do handwavy analysis of your backed-out λ values:
It’s notable that Terminal-Bench showed no swarm benefit in the paper above, and that’s also the lowest λ you found. The other OpenAI 16-agent cases were also using Ultra mode, which by my understanding is more of a traditional hierarchical multi-agent system than the freely-communicating systems that Anthropic and OpenAI are moving towards. So we might expect this to translate into:
Meaning superlinear returns to adding more agents, given fixed test-time compute. That is, rather than more parallel agents giving you diminishing returns to capabilities, more parallel agents give you increasing returns to capabilities, at least within some useful bounds.
Or equivalently, four people spend 10 hours on the problem within a week vs. one person spends 40 hours within a week.
Maybe a naive question, I’m having trouble understanding how λ > 1 could be possible in principle; if a 10 agent swarm can do 11 times the work, then could I not improve my single agents by serializing this parallelization? (I think I’m just parroting some bog-standard argument in parallel computing.) That is, have my single meta-agent act as agent 1 for a while, then agent 2 for a while, and so on, the same way a single-core CPU simulates multi-threading.
Initially I thought that a swarm might win out against the above setup in efficiency by saving on the cost of context switching, but I would guess that loading up a new agent has the at least the same (and possibly higher) cost as context switching an existing one?
No that’s a great question, thanks! The paper I discussed above actually agrees with you, that’s why their model of how this works caps λ at 1.
My counterargument is that this is true for an idealized serial agent. We can refer to the human example again for intuition:[1]
We can see that λ > 1 could be true in a couple core scenarios:
The 4 people have different skillsets that complement each other, while the 1 person has just one skillset.
All 5 people have the same skillset, but they take different approaches due to random chance/coordination/initial conditions/experience, and one of those approaches is more fruitful. Assuming a large number of possible approaches, it’s more likely the team finds a better one.
With optimal approach switching, the 1 person could attempt four different approaches in the same amount of time, making it even odds, but that isn’t really how people work in practice. Idealized vs. real.
You can also see here why agent homogeneity is so problematic.
Re: efficiency on context switching, I think you’re right that swarm has no efficiency there, optimally. But of course one of agents’ main flaws (and humans’ main flaws?) is getting too attached to the current context/approach/belief.[2]
Reformulated to use version from footnote of previous comment, which in retrospect I prefer to avoid confusion about task speedup via parallelization, which is a separate matter.
Though you could just employ a setup that wipes the serial agent’s context entirely, and denies it access to previous work. Luckily we can’t do that to humans!
I see, I agree that it’s possible that a particular AI design easily gets caught in its context and periodically wiping it like an anterograde amnesiac might actually make it do better.
I guess it feels like if we ever had a model that was λ > 1 that would signal some serious inadequacy of the model that feels not hard to RL out?
A good theoretical question! If you have a set of humans with λ > 1, is that a sign of serious inadequacies in some or all of them, which ought to be addressed? Possibly!… if you want to optimize for task success, anyway.[1]
Thinking more about our context-switching serial agent:
On tasks where sharing intermediate results is very valuable (ex. hard research tasks), you really want the “trained to message other agents about intermediate results, and know how to trust/accept/reject those messages” behavior Anthropic/OpenAI are currently pursuing. Theoretically you could implement that by just having the serial agent send messages to its future selves. But the RL to teach that is multi-agent system RL! And the easiest implementation is real-life swarms.[2]
To put that another way, I expect we’ll have agents trained to send messages in a multi-agent system in an effective way. If you run a practical test with just one of those agents that has no one to message, against the multi-agent system, then you can get λ > 1. There will be a theoretical setup in which a serial agent matches the multi-agent system performance, but it won’t be used in practice.[3]
Thinking more about heterogeneity:
There’s an interesting path where λ > 1 in variants of a single model. A possible future: we have some basic continual learning using overnight weights-updating for agents working different jobs/roles, a more advanced version of Anthropic’s current long-running agent identities setup. There is no single agent that knows everything and that could substitute for all of them, because getting all of that context into the weights crowds model identity/skills/context/J-space in some way that makes them less effective.[4]
That all said, you’ve clarified my thinking here and I’ve updated down on λ > 1 thanks to this conversation.
In the limit, this almost feels like talking about whether a single grand unified consciousness is superior to a civilization, or an ASI singleton vs. ASI swarms, though we would have to carefully decompose “superior” there.
Though you could implement this in your serial agent by giving it the ability to end its current session with a “wait for reply/new message” tool. There might be some tricky caching issues with that but perhaps surmountable.
What does this mean for forecasting? I think the practical effects here mean that the practical speedup can still be equivalent to λ > 1. That is, parallelism can deliver increasing marginal returns (again, at least within some bounds on N agents) on forecasted available compute compared to the old pre-swarm baseline. However, you could just adjust the baseline to the ideal multi-agent-system-simulating serial agent, which clamps λ ≤ 1.
But it’s plausible no such limit exists in practice, or that more heterogeneity than this is needed, aka you need substantially different training runs to really get near-human variety.
Interesting. My initial intuition was that an agent which randomizes u.a.r. over all permutations of approaches should perform at least as well as the swarm. But this isn’t true! If we have 4 approaches, 3 of which lead to an hour of wasted time and 1 which leads to instant success, then a swarm of 4 agents will always be instantly successful. While a single agent will waste 0, 1, 2 or 3 hours, for an average of 1.5h wasted.
This is only theoretical of course. The example is chosen to be very extreme. Also, in real life we have communication overhead, the need for a “main” agent to verify the results, etc. But I think it’s a good intuition pump.
with randomized algos like SAT solvers, you can simulate early stopping of parallel runs by a round robin on a single thread, so in principle and in the limit you shouldn’t be able to get superlinear scaling just from theory.. sure, in practice, on actual hardware and software, there will be sweet spots a la L1 cache physical distance from the logic core, or the whole reason GPUs exist … but I wouldn’t count on the sweet spot that will exist for ~10 agents to then scale to ~10 million agents with a failure of competitive pressure to find improvements to single/few-agent performance in the next month
Anything constrained by total working memory is a candidate
I think an ideal serialized agent wouldn’t be more constrained by working memory than a swarm would be. It’s not handling all the context all at once, it’s context-switching. It can emulate the full swarm (and even messaging behavior!) that way, see more discussion of that here.
But then again, we saw that kind of logic break down in practice in multi-threading on CPUs because of caching. In theory, a single-core CPU can emulate an N-core CPU in N× as much time. In practice, constantly swapping out the whole CPU L1/L2 cache wastes time, so a single-core CPU takes longer. Or rather, an N-core CPU is more than N× better; λ > 1.
So this comes back to the discussion on context-switching costs. The simple case is that you switch once per swarm agent the serial agent is simulating, and in that case caching costs can be equal. But the organized swarm case requires switching most every time the serialized agent wants to send or receive a message, which would be a lot more cache-swapping than running the agents in parallel… and with a lot bigger caches than CPUs have. So ex. if you have a nearly-full context window of 900k tokens, and you send a message after every 5k tokens of reasoning before swapping out the whole cache, you’re in trouble![1]
However even then, I suspect this might be less relevant than heterogeneity.
I don’t think clever KV-cache swapping is gonna save you here either when you’re trying to simulate 10,000 agents. Sorry Deepseek!
Agents are only now starting to be trained to cooperate with each other and self-organize.
In my experience today’s agents frequently make obvious coordination mistakes and fail to use good theory of mind or have high-trust patterns of communication when prompting each other. They feel like they are optimized for doing everything themselves and at best delegating very specific tasks to subagents.
I haven’t looked into SEC-Bench Pro, but if you take a benchmark that can be done pretty well by a single agent, it makes sense that it’s not that parallelizable, but things like “running a large corporation” clearly can’t be done without some mechanism to divide attention. For LLMs, things work somewhat differently, since you can clone agents and share context more cheaply, but it is definitely not going to make sense to chuck all the inputs of your company into a single smart agent’s context and have them make every decision.
Also, time does matter. If an agent could have solved Navier Stokes by carefully maintaining a large file system for itself and working on each subproblem one at a time, maybe it could produce the whole Lean proof in ~5 years. There are many kinds of problems where that would not be a success.
Civilization basically just is effective mechanisms of dividing labor so that a bunch of people can focus their finite attention/context windows on doing one thing well at a time. It would not work if you only had one guy trying to do everything at once, even if that guy were really competent.
I think humans kind of suck at cooperating with each other and self-governing, compared to what is possible. The difference between the least effective teams of humans and the most effective teams, comprised of all the same people with different communication norms, already spans orders of magnitude, but even the most effective humans are more limited in trust and communication bandwidth than LLMs could/likely soon will be.
Thanks Julian, that’s very thoughtful. I’ve now read your post on this and I thought it was good.
I’m definitely less bullish than you are about getting λ>1, though it may be possible, especially in cases where agent diversity is important (and where that can be implemented somehow) or where it is hard for a single agent to try out lots of different approaches successively without them contaminating each other.
You may be interested to know that Phil Trammell is thinking about something similar to increasing λ in the essay below. In fact he sees understanding parallelisability as key to understanding the prospects on an intelligence explosion:
https://philiptrammell.com/static/Parallelizability.pdf
Ah, a much more formal take that decomposes the relevant factors, I gather. I’ll look into that, thanks!
Edit: also FYI I haven’t looked deeply yet, but Opus 5.5 System Card would seem to still show λ in the 0.5 to 0.7 range, like I imagine you would predict.
Not-too-important Addendum: another paper I saw today: “Self-Organizing Agent Teams Learn to Reason Together”. The headline relevant result here is:
This sounds like λ > 1, but I think it’s more an exercise in caution on skimming arXiv preprints, as I don’t think this is good evidence:
Their linearization test is executing the strategies developed by the multi-agent, multi-model team of 3. Not apples to apples. A multi-agent, single-model team of 3 does worse* (though frustratingly it’s also executing given strategies). Performance gains may partly come from heterogeneity.[1]
Though Table 4 tries to demonstrate they come at least partly from the self-organized team structures.
*Except for one benchmark (BIG-Bench Extra Hard) where it does better for some reason.
Also “does better/worse” is quite possibly mostly inside confidence intervals here… which aren’t reported. (!)
“swarm” sizes here are just 3, and models are old (ex. Sonnet 4). Sample size also kind of small.
Compute-matching is loose; no token counts given, word “approximately” is used.
Headline result applies only to some of the benchmarks they tested.
On the whole I’m not comfortable updating much on this evidence. (also in terms of self-organizing teams results, unfortunately these folks were scooped by a more interesting and relevant result from Anthropic’s Opus 5.5 System Card, Ctrl-F for “Emergent team structures”)
You know I’ve found it useful myself to compare/contrast multiple LLMs’ results on programming tasks. I wonder if there might be gains to fine-tuning a lot of variants of a single frontier model, or if you really need different pretrains to get value.