LLM-as-a-Judge
An LLM that grades another model's output
Giving an LLM a rubric and having it score another model's output (or its own). A scalable way to measure quality that can't be checked in code but can be described in a written criterion; but it has to be calibrated against human labels first.
LLM-as-a-judge means having an LLM do the scoring in evals. The work that popularised the term is Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023). The paper defines three variants:
- Pairwise comparison: a question and two answers; the judge picks the better one or declares a tie.
- Single answer grading: the judge scores one answer directly.
- Reference-guided grading: the judge also gets a reference solution.
Per the paper, a strong judge like GPT-4 agreed with human preferences over 80% of the time, the same level as agreement between humans. The same work lists documented biases:
- Position bias: the verdict can change with the order of the answers.
- Verbosity bias: preferring the longer answer even when it isn't better.
- Self-enhancement: the judge favouring answers it generated. Zheng et al. say they couldn't establish it conclusively; Panickssery et al. (2024) reported that models can recognise their own text and that this is correlated with self-preference.
- Limited reasoning: a judge can be misled by a wrong answer even on a question it could solve on its own.
A judge is a measuring instrument: measure it on a human-labelled sample first, then scale it.
Like adding an assistant judge to a competition panel. The assistant never tires and can watch thousands of performances; but on day one it sits next to the head judge, they score the same performances, and the scores are compared. If the assistant always favours whoever goes on stage first, or gives long performances high marks automatically, it isn't left alone. And it never grades its own students.
Whether a support bot's replies "follow the refund policy" can't be checked in code. The team:
- Has a domain expert label 200 replies pass/fail and splits the sample in two: one half to develop the judge prompt, the other to measure it.
- Picks a judge from a different model family than the model being evaluated; the rubric is narrow and binary: "Does the reply promise a refund beyond 30 days? Write your reasoning first, then only PASS or FAIL."
- Checks the confusion matrix and Cohen's kappa on the held-out half; reading the cases where judge and human disagree, it fixes an ambiguous sentence in the rubric.
- When comparing two prompt versions, asks about each pair twice with the answers swapped, and only counts a version as the winner if it wins in both orders.
JUDGE_PROMPT = """Below are a user question and two assistant answers.
Which one answers the question more accurately and helpfully?
Length alone is not a quality criterion.
Write a short rationale first; on the last line write only A, B or TIE.
[Question]
{question}
[Answer A]
{a}
[Answer B]
{b}"""
def judge(question, a, b) -> str:
... # call the judge model, return the last line ("A" | "B" | "TIE")
def pairwise(question, answer_1, answer_2) -> str:
first = judge(question, answer_1, answer_2) # 1 = A
second = judge(question, answer_2, answer_1) # 1 = B
if first == "A" and second == "B":
return "answer_1"
if first == "B" and second == "A":
return "answer_2"
return "tie" # different verdicts in the two orders = tie- When the criterion can be written down but not coded: tone, faithfulness, policy compliance, helpfulness
- Comparing two prompt or model versions over many examples
- Monitoring quality asynchronously on a sample of live traffic
- When human review doesn't scale but agreement with humans can be measured
- When the answer can be checked in code (JSON schema, exact match, unit test): code is faster, cheaper and repeatable
- For decisions, if the judge's agreement with human labels has never been measured
- On expert correctness questions the judge itself can't answer, without a reference
Ignoring position bias
Wang et al. (2023) showed they could flip results just by reordering answers: with ChatGPT as the evaluator, Vicuna-13B 'beat' ChatGPT on 66 of 80 queries. In pairwise comparisons, ask about each pair in both orders.
Using the evaluated model as its own judge
Self-preference is a risk: Panickssery et al. (2024) found a linear correlation between self-recognition ability and the strength of self-preference bias. Use a judge from a different model family where possible.
Looking only at percent agreement
If most labels are 'pass', a judge that passes everything also shows high agreement. Look at the confusion matrix, true positive/negative rates and a chance-corrected measure such as Cohen's kappa.
Tuning and measuring on the same examples
Measuring the judge on the examples you tuned its prompt on teaches it those examples. Keep a separate, unseen sample.
Broad, multi-criteria rubrics
Scores like 'overall quality from 1 to 10' are hard to calibrate. Anthropic's docs recommend clear rubrics and specific outputs such as 'only correct or incorrect'; a separate binary judge per criterion is usually easier to measure.