AI Atlas
Intermediate · ~2 min read #llm-as-a-judge #evals #evaluation

LLM-as-a-Judge

An LLM that grades another model's output

Giving an LLM a rubric and having it score another model's output (or its own). A scalable way to measure quality that can't be checked in code but can be described in a written criterion; but it has to be calibrated against human labels first.

LLM JUDGE · SWAP THE ORDER, MEASURE AGAINST HUMANSPAIRWISE · ASKED TWICEABJUDGEA winsBAJUDGEA winsswapSame answer in both orders → winnerChanged with order → tieDOCUMENTED BIASESpositionorder flips the verdictverbositylonger ≠ betterself-preferencefavours its own outputCALIBRATION · HUMAN × JUDGETPFNFPTNhuman ✓human ✗judge ✓judge ✗kappaTPR/TNRA judge is an instrument: measure it on a human-labelled sample, then scale it
Definition

LLM-as-a-judge means having an LLM do the scoring in evals. The work that popularised the term is Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023). The paper defines three variants:

  • Pairwise comparison: a question and two answers; the judge picks the better one or declares a tie.
  • Single answer grading: the judge scores one answer directly.
  • Reference-guided grading: the judge also gets a reference solution.

Per the paper, a strong judge like GPT-4 agreed with human preferences over 80% of the time, the same level as agreement between humans. The same work lists documented biases:

  • Position bias: the verdict can change with the order of the answers.
  • Verbosity bias: preferring the longer answer even when it isn't better.
  • Self-enhancement: the judge favouring answers it generated. Zheng et al. say they couldn't establish it conclusively; Panickssery et al. (2024) reported that models can recognise their own text and that this is correlated with self-preference.
  • Limited reasoning: a judge can be misled by a wrong answer even on a question it could solve on its own.

A judge is a measuring instrument: measure it on a human-labelled sample first, then scale it.

Analogy

Like adding an assistant judge to a competition panel. The assistant never tires and can watch thousands of performances; but on day one it sits next to the head judge, they score the same performances, and the scores are compared. If the assistant always favours whoever goes on stage first, or gives long performances high marks automatically, it isn't left alone. And it never grades its own students.

Real-world example

Whether a support bot's replies "follow the refund policy" can't be checked in code. The team:

  1. Has a domain expert label 200 replies pass/fail and splits the sample in two: one half to develop the judge prompt, the other to measure it.
  2. Picks a judge from a different model family than the model being evaluated; the rubric is narrow and binary: "Does the reply promise a refund beyond 30 days? Write your reasoning first, then only PASS or FAIL."
  3. Checks the confusion matrix and Cohen's kappa on the held-out half; reading the cases where judge and human disagree, it fixes an ambiguous sentence in the rubric.
  4. When comparing two prompt versions, asks about each pair twice with the answers swapped, and only counts a version as the winner if it wins in both orders.
Code examples
Pairwise comparison with swapped order (Zheng et al.'s conservative approach) python
JUDGE_PROMPT = """Below are a user question and two assistant answers.
Which one answers the question more accurately and helpfully?
Length alone is not a quality criterion.
Write a short rationale first; on the last line write only A, B or TIE.

[Question]
{question}

[Answer A]
{a}

[Answer B]
{b}"""

def judge(question, a, b) -> str:
    ...  # call the judge model, return the last line ("A" | "B" | "TIE")

def pairwise(question, answer_1, answer_2) -> str:
    first = judge(question, answer_1, answer_2)   # 1 = A
    second = judge(question, answer_2, answer_1)  # 1 = B
    if first == "A" and second == "B":
        return "answer_1"
    if first == "B" and second == "A":
        return "answer_2"
    return "tie"  # different verdicts in the two orders = tie
When to use
  • When the criterion can be written down but not coded: tone, faithfulness, policy compliance, helpfulness
  • Comparing two prompt or model versions over many examples
  • Monitoring quality asynchronously on a sample of live traffic
  • When human review doesn't scale but agreement with humans can be measured
When not to use
  • When the answer can be checked in code (JSON schema, exact match, unit test): code is faster, cheaper and repeatable
  • For decisions, if the judge's agreement with human labels has never been measured
  • On expert correctness questions the judge itself can't answer, without a reference
Common pitfalls

Ignoring position bias

Wang et al. (2023) showed they could flip results just by reordering answers: with ChatGPT as the evaluator, Vicuna-13B 'beat' ChatGPT on 66 of 80 queries. In pairwise comparisons, ask about each pair in both orders.

Using the evaluated model as its own judge

Self-preference is a risk: Panickssery et al. (2024) found a linear correlation between self-recognition ability and the strength of self-preference bias. Use a judge from a different model family where possible.

Looking only at percent agreement

If most labels are 'pass', a judge that passes everything also shows high agreement. Look at the confusion matrix, true positive/negative rates and a chance-corrected measure such as Cohen's kappa.

Tuning and measuring on the same examples

Measuring the judge on the examples you tuned its prompt on teaches it those examples. Keep a separate, unseen sample.

Broad, multi-criteria rubrics

Scores like 'overall quality from 1 to 10' are hard to calibrate. Anthropic's docs recommend clear rubrics and specific outputs such as 'only correct or incorrect'; a separate binary judge per criterion is usually easier to measure.