LLM-as-a-Judge: Using AI to Evaluate AI Output at Scale

Human evaluation of LLM outputs is the gold standard — but it doesn't scale. When you're generating thousands of responses per day, you need automated quality assessment. LLM-as-a-judge uses a capable language model to evaluate the outputs of another, providing scalable quality scores without human bottlenecks.

How LLM-as-a-judge works

You pass the original prompt, the model's response, and an evaluation rubric to a judge model. The judge returns a score, a verdict, or a structured assessment:

judge_prompt = """You are evaluating a customer support response.

Original customer question: {question}
Support response: {response}

Rate the response on these criteria (1-5):
- Accuracy: Does it correctly address the issue?
- Completeness: Does it fully resolve the query?
- Tone: Is it professional and empathetic?

Return ONLY valid JSON:
{{"accuracy": N, "completeness": N, "tone": N, "pass": true/false, "reason": "..."}}"""

result = await judge_client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": judge_prompt.format(
        question=original_question,
        response=model_response
    )}],
    response_format={"type": "json_object"}
)

Pairwise comparison vs absolute scoring

Two main evaluation modes:

• Absolute scoring — rate a single response on a scale. Fast but inconsistent across judge calls.
• Pairwise comparison — show the judge two responses and ask which is better. More reliable, used by LMSYS Chatbot Arena. Slower and costs 2x the tokens.

Pairwise comparison correlates better with human preferences. Use it when you're comparing models or prompt variants; use absolute scoring for production monitoring.

Known biases to account for

• Verbosity bias — judges consistently prefer longer responses, even when shorter is objectively better
• Self-preference bias — GPT-4 rates GPT-4 outputs higher; Claude rates Claude outputs higher. Use cross-model judging.
• Position bias — in pairwise evals, the first response shown gets a slight advantage. Randomize order and average both orderings.
• Authority bias — responses that sound confident are rated higher regardless of accuracy

Best practices

• Use a different model family as judge than the model being evaluated
• Include explicit rubrics — vague prompts like "is this good?" produce noisy scores
• Calibrate your judge against human ratings on 50-100 examples before trusting it
• Use response_format: json_object to get structured, parseable scores
• Track judge agreement rate over time — it drifts as models update

Cost of LLM-as-a-judge at scale

Running a GPT-4o judge on every production response is expensive. A 500-token prompt + 200-token judgment = 700 tokens × $0.005/1K = $0.0035 per eval. At 100K evals/day that's $350/day just for evaluation. Strategies to reduce this:

• Sample 1-5% of production traffic for eval instead of 100%
• Use a cheaper judge (GPT-4o-mini, Claude Haiku) for initial triage, escalate to GPT-4o only for borderline cases
• Batch evals during off-peak hours