Term
LLM-as-a-Judge
LLM-as-a-Judge is the practice of using a language model as an evaluator — it compares responses from other models or scores outputs against given criteria.
LLM-as-a-Judge — explained in detail
Instead of having humans grade every model output, a second, often stronger LLM takes the role of referee. It receives a task, the response under review, and a scoring rubric — for example, “rate factual faithfulness from 1 to 5” or “which of the two responses is better?”. Typical variants: single grading (one response, absolute score), pairwise comparison (two responses, pick a winner), and reference-based judging against a gold answer.
Example / Practical use
In RAG pipelines, judges automatically score faithfulness, answer relevancy, and context precision across thousands of examples. Frameworks like RAGAS, DeepEval, and Promptfoo run judges in production. Chatbot Arena also uses a judge model as a pre-filter. Rule of thumb: a judge should be at least one model class stronger than the system under test, otherwise its reliability drops sharply.
Distinction from related terms
LLM-as-a-Judge is a tool inside an eval pipeline, not a replacement for human evaluation — judges inherit training-data bias and tend toward position bias (favoring the first response) and length bias (favoring longer answers). Mitigations: calibrate the judge, use a jury of multiple models, or validate against human samples. Distinction: a classic benchmark has objective gold answers, while a judge can also score open-ended responses.
Discover more
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossaryGround Truth
A reference taken to be correct, against which model outputs are measured. In AI code analysis it is a piece of code with deliberately injected, known defects, so you can measure how many real problems a model actually finds.
EncyclopediaAgent Evaluation — Task Success Rate, Eval Harnesses, Benchmarks
How to evaluate AI agents: eval harnesses, task success rate, benchmarks like GAIA and Tau-Bench, LLM-as-a-judge — and how it differs from pure model quality.