LLM Evaluation: Building Evals and Regression Tests That Actually Catch Production Failures

Your chatbot passed every test you threw at it. You ran a dozen prompts, the answers looked reasonable, you shipped it. Three weeks later, support tickets start piling up — the bot is confidently telling customers something that isn’t true. Nothing in your codebase changed. The model behind the API did.

That’s the failure mode that makes LLM-powered features different from everything else you’ve shipped. A RAG pipeline can retrieve exactly the right documents and still generate an answer that contradicts them outright — the retrieval step succeeds, the generation step quietly fails, and nothing in your logs screams “bug” the way a stack trace would.

If you’ve shipped an LLM feature and you’re now wondering how to actually know whether it works — not “feels like it works,” but measurably, repeatably works — you need LLM evaluation. This article gives you a working system: a definition that explains why your instincts from traditional testing don’t transfer, a two-track architecture (offline lab, online floor), the specific metrics worth tracking, and a worked harness you can adapt to your own stack.

What LLM evaluation actually is, and why unit tests don’t cut it

LLM evaluation is the practice of measuring how well an LLM or LLM-powered application performs against defined quality criteria — running the system against a set of test cases, scoring the outputs through automated or manual methods, and tracking that performance over time so you can detect improvements or regressions.

The reason you can’t just write assert response == expected_output is structural, not a matter of discipline. Unlike deterministic software, an LLM can generate a different output each time you run it on the exact same input, because generation is probabilistic — there is no single correct output to pin a test against. At any temperature above zero, the same prompt can legitimately produce several valid-but-different phrasings, and a small change in wording can sometimes cascade into a very different output altogether. A traditional unit test assumes a fixed, correct answer exists. Evals assume a distribution of acceptable answers exists, and your job is to measure where new outputs fall in that distribution.

It helps to think of this as a manufacturing-quality problem rather than a software-correctness problem. A RAG pipeline, for instance, is a probabilistic assembly line — retrieved documents feed into a foundation model, and the output is assembled, not computed. Running ten prompts and eyeballing whether the answers look right — the “vibe check” — might be fine for a tool only you use, but it collapses the moment the feature is in front of real customers at scale, because you’ve sampled almost none of the input space and you have no record to compare against later.

The four jobs evals are actually doing

Before building anything, it’s worth being explicit about what evaluation is for, because that shapes what you build. Evals serve four distinct purposes: establishing a performance benchmark to inform design decisions, confirming outputs stay consistent and reliable, surfacing insight that guides iterative improvement, and enabling regression testing whenever you change a prompt, swap a model, or alter the pipeline.

Notice that only one of those four — regression testing — is about catching breakage. The other three are about understanding and improving the system over time. If your “eval strategy” is a single pass/fail gate you built once and never revisit, you’re only covering a quarter of the job.

Model evals vs. product evals: you’re not grading the LLM, you’re grading your feature

A common confusion: standardized model benchmarks tell you about raw model capability, not about whether your feature works. Model evaluation measures an LLM’s general capabilities against standardized benchmark tasks, while product evaluation measures whether your specific system solves the real problem your users have — and a model can score well on public benchmarks while still performing poorly on your domain-specific task.

This matters practically: if you’re choosing between two models for a support chatbot, their published benchmark scores are a weak proxy for “will this model correctly answer questions about our refund policy.” You need evaluation data built from your domain, not someone else’s leaderboard.

Two tracks: the offline lab and the online floor

The core architectural idea — and the one that should reshape how you organize eval work — is splitting evaluation into two tracks with different jobs.

Offline evaluation runs during development, before anything reaches a user. It’s testing against a curated, known dataset — checking a prompt rewrite, a model swap, or a retrieval change against inputs you’ve already classified and scored. It plays the role unit and integration tests play in traditional software: catching problems before they ship.

Online evaluation runs against live production traffic. You sample real requests, score them asynchronously, and track quality metrics alongside your usual operational metrics like latency and error rate. This is what catches what offline testing structurally cannot: novel queries you never anticipated, shifts in the input distribution, and gradual model drift.

The split maps cleanly onto the manufacturing framing: an offline laboratory runs a permanent golden dataset of hundreds of diverse queries, source documents, and ground-truth answers every time the architecture changes, measuring the statistical variance in response quality — and an online production floor uses small, efficient evaluator models to score live interactions in flight, ideally before a hallucination reaches the user.

You can also think of this as three stages rather than two: offline evaluation builds a bank of anticipated production scenarios before deployment; inline evaluation/guardrails run lightweight checks (toxicity, format validity) at runtime on critical steps so a bad response can be swapped out before the user sees it; and online evaluation samples live traffic through the same evaluators you built offline. The middle stage — guardrails — is what separates “we’ll catch it in the next eval cycle” from “we caught it before the user did.”

Concrete failure modes these two tracks are built to catch

Two failure patterns show why both tracks matter, and why neither one alone is sufficient:

A customer support chatbot can perform accurately for weeks, then start producing incorrect information after the underlying model provider pushes an update — nothing in your code changed, but the behavior did. Only online evaluation catches this, because your offline golden dataset was last run before the update happened, and nobody re-runs it on a schedule unless drift detection prompts them to.

A RAG pipeline can retrieve exactly the right source documents and still generate an answer that contradicts them — the retrieval component passes, the generation component silently fails. This is why component-level evaluation matters: if you only measure end-to-end correctness, you’ll know something is wrong without knowing whether to debug your retriever or your prompt.

Component-level vs. end-to-end evaluation

Component-level evaluation isolates one part of the pipeline — retrieval quality in a RAG system, tool-selection accuracy in an agent — while end-to-end evaluation measures whether the complete system achieved the user’s actual goal. When end-to-end scores drop, component-level scores are what let you localize the cause instead of guessing.

In practice: instrument your retriever, your tool-calling layer, and your final generation step as separately scoreable stages, even if you also report one end-to-end quality number to stakeholders.

RAG-specific metrics: the three that matter most

For a RAG pipeline specifically, three metrics give you the clearest picture, because they separate retrieval failure from generation failure instead of conflating them:

Context Precision — did retrieval actually find the right information for the query?
Faithfulness — is the generated answer grounded in the retrieved documents, or is the model fabricating facts not present in them?
Answer Relevance — does the response actually address what the user asked?

This maps to the two-stage nature of RAG itself: retrieval evaluation asks whether relevant documents were found, and generation evaluation asks whether the answer produced from those documents is good. A RAG pipeline can score well on Context Precision and poorly on Faithfulness simultaneously — that combination is the exact “right docs, wrong answer” failure described above, and it’s invisible unless you measure the two separately.

If you’re building the retrieval layer yourself — for example, a pgvector-based pipeline — these metrics are the natural test suite to run every time you touch the embedding model, chunking strategy, or similarity threshold.

Agent evaluation: judging a trace, not just an answer

Agents complicate evaluation further because the thing you’re scoring isn’t a single response — it’s a sequence of decisions. Relevant dimensions include tool-selection accuracy (did it call the right tool?), task completion rate (did it finish the job?), and execution efficiency (did it take a reasonable path, or loop and burn tokens?) — all of which require inspecting the full multi-step trace, not just the final message.

Practically, this means your eval harness for an agent needs to log the entire sequence of tool calls and intermediate reasoning, not just input/output pairs, because a correct final answer can still hide an inefficient or risky path to get there.

Fine-tuned models: watch for catastrophic forgetting

If you fine-tune a model for a narrow task, your eval suite needs a second job beyond measuring target-task performance: confirming the model hasn’t lost general capability it had before — a failure mode generally known as catastrophic forgetting. Run your fine-tuned model against both your target-task test set and a baseline-performance set representing its prior general behavior, and compare both before and after each fine-tuning run.

If your system routes requests across multiple models or chains (cheap model for simple queries, stronger model for complex ones; one agent for billing questions, another for technical ones), add evaluation specifically for the routing decision itself. A model can perform perfectly once selected and still produce a bad user outcome because the router sent the query to the wrong place. Track routing accuracy as its own metric, separate from the downstream model’s output quality — otherwise a misrouting bug looks identical to a model-quality bug in your dashboards.

Safety and responsible AI checks

Quality isn’t only about correctness. A mature eval suite also screens for toxicity, bias, privacy exposure (is the model leaking something it shouldn’t), and robustness against prompt injection — these belong in both your offline test bank and your online guardrail layer, since a jailbreak attempt is exactly the kind of input your golden dataset probably didn’t anticipate.

Automated evaluation, human review, and LLM-as-a-judge

You won’t scale evaluation by hand-reviewing every output, but you also can’t fully automate judgment calls that require nuance. The practical answer combines both: automated evaluation (programmatic checks, regex, exact-match, proxy metrics) handles scale and repeatability, while human-in-the-loop review handles subjective quality judgments, context-dependent nuance, and high-stakes validation.

Sitting between the two is LLM-as-a-judge — using one LLM to evaluate another’s outputs — which is one of the most widely used evaluation methods in practice today. It scales better than human review and captures more nuance than a regex, but it introduces its own failure surface: a judge model has its own biases and blind spots, so treat its scores as a strong signal worth sampling and spot-checking with humans, not as unquestionable ground truth.

Reference-based vs. reference-free evaluation

These two modes answer different questions, and conflating them leads to the wrong tool for the job. Reference-based evaluation compares an output against a known correct answer, and is the right choice for experimentation, regression testing, and deliberate stress-testing where you’ve defined what “correct” looks like. Reference-free evaluation judges an output on its own merits without a ground truth to compare against, and is the natural fit for production monitoring, guardrails, and open-ended conversational scenarios where no single correct answer exists.

Your offline golden dataset is naturally reference-based — you wrote the ground truth. Your online production scoring is naturally reference-free — you’re judging live, novel traffic you have no prewritten answer for.

The quality metrics taxonomy

Pulling the above together, here’s the practical set of metrics worth tracking, organized by what they actually measure:

Category	What it measures	Typical technique
Factual consistency / faithfulness	Whether the response accurately reflects source context — critical for RAG	Reference-based or LLM-as-a-judge
Response quality	Coherence, fluency, helpfulness, completeness, conciseness	LLM-as-a-judge, human review
Safety & ethics	Toxicity, bias, privacy exposure	Automated classifiers, guardrails
Relevance	Contextual relevancy, answer relevancy — especially in multi-turn dialogue	LLM-as-a-judge
Robustness	Handling typos, ambiguous phrasing, adversarial prompts	Targeted adversarial test cases

For more mechanical tasks, simpler metrics apply directly: context recall for RAG retrieval, exact match for classification, JSON validators for structured extraction, and semantic similarity for creative or open-ended output. Use the cheapest metric that’s actually valid for the task — don’t reach for an LLM judge to check whether a field parses as valid JSON.

Building your golden dataset

A golden dataset is the foundation everything else depends on, and it’s worth treating as a first-class, version-controlled asset rather than a notebook you wrote once. Build it from question-answer pairs, representative edge cases, and — where the domain demands it — review by people who actually know the subject matter, not just engineers guessing at what “correct” looks like. For a RAG system specifically, aim for hundreds of diverse queries paired with their source documents and ground-truth answers, large enough that you can measure statistical variance in response quality rather than reacting to single anecdotes.

Don’t build this dataset once and freeze it. Mature practice treats evaluation as a continuous loop: scrub live telemetry of anything private, then feed flagged or borderline interactions back into the offline golden dataset so it keeps representing the traffic you actually see, not just the traffic you imagined at launch.

A worked regression-testing pattern

Here’s a minimal pattern for a FastAPI-based service, adaptable to either the OpenAI or Anthropic API. The idea: define your golden dataset as structured test cases, run every candidate prompt/model configuration against all of them, score with a mix of deterministic checks and an LLM judge, and diff against your last recorded baseline.

# eval_harness.py
from dataclasses import dataclass
from typing import Callable
import json

@dataclass
class TestCase:
    id: str
    query: str
    context_docs: list[str]
    expected_answer: str | None  # None for reference-free cases
    tags: list[str]              # e.g. ["rag", "edge-case", "multi-turn"]

@dataclass
class EvalResult:
    test_id: str
    faithfulness: float
    answer_relevance: float
    passed: bool
    notes: str

def load_golden_dataset(path: str) -> list[TestCase]:
    with open(path) as f:
        rows = json.load(f)
    return [TestCase(**r) for r in rows]

def judge_faithfulness(answer: str, context_docs: list[str], judge_call: Callable) -> float:
    # judge_call wraps your chosen LLM-as-a-judge prompt against
    # either the OpenAI or Anthropic API — return a 0-1 score
    prompt = (
        "Score 0-1: is every claim in ANSWER directly supported by CONTEXT?\n"
        f"CONTEXT:\n{context_docs}\n\nANSWER:\n{answer}"
    )
    return judge_call(prompt)

def run_eval_suite(
    system_under_test: Callable,
    dataset: list[TestCase],
    judge_call: Callable,
    faithfulness_threshold: float = 0.8,
) -> list[EvalResult]:
    results = []
    for case in dataset:
        answer = system_under_test(case.query, case.context_docs)
        faithfulness = judge_faithfulness(answer, case.context_docs, judge_call)
        relevance = judge_call(f"Score 0-1 relevance of ANSWER to QUERY:\n{case.query}\n{answer}")
        passed = faithfulness >= faithfulness_threshold
        results.append(EvalResult(case.id, faithfulness, relevance, passed, notes=""))
    return results

def diff_against_baseline(current: list[EvalResult], baseline: list[EvalResult]) -> list[str]:
    regressions = []
    baseline_map = {r.test_id: r for r in baseline}
    for r in current:
        prior = baseline_map.get(r.test_id)
        if prior and r.faithfulness < prior.faithfulness - 0.1:
            regressions.append(f"{r.test_id}: faithfulness dropped {prior.faithfulness:.2f} -> {r.faithfulness:.2f}")
    return regressions

Wire run_eval_suite into CI the same way you’d wire any other test suite: run it on every PR that touches a prompt file, a retrieval parameter, or the model name string, and fail the build if diff_against_baseline returns regressions above your tolerance. Store each run’s results alongside the git commit hash so you can trace exactly which prompt or model change caused a quality drop later.

For online scoring, the same judge_call and metric functions apply — the difference is you’re sampling a percentage of live requests asynchronously (so you’re not adding judge-model latency to the user-facing response) and writing results to a dashboard rather than failing a CI step.

Putting it together

Evaluation isn’t a milestone you complete before launch — it’s infrastructure you maintain for as long as the feature exists, because the model underneath you, the traffic hitting it, and the prompts wrapping it will all keep changing. Build the offline lab first: a golden dataset, component and end-to-end metrics, and a CI gate that fails on regression. Then build the online floor: sampled live scoring with the same metrics, drift alerts, and a scrubbing pipeline that feeds interesting failures back into your golden dataset. Do both, and a model update three weeks from now becomes a dashboard alert instead of a support ticket.

FAQ

Is LLM-as-a-judge reliable enough to trust on its own? It’s one of the most widely used methods for scoring LLM output at scale, but it’s not infallible — pair it with periodic human review, especially for high-stakes outputs, rather than treating judge scores as ground truth.

How big should a golden dataset be? There’s no universal number, but aim for enough diverse queries, documents, and ground-truth answers that you can see statistical variance in scores rather than reacting to one-off anecdotes — for a RAG system that typically means hundreds of cases, not a handful.

Do I need both offline and online evaluation, or can I pick one? They catch different things: offline evaluation catches regressions before deployment using known inputs, while online evaluation catches drift, novel queries, and distribution shifts that only show up in live traffic — skipping either leaves a real gap.

What’s the difference between evaluating a model and evaluating my product? Model evaluation measures general LLM capability on standardized benchmarks; product evaluation measures whether your specific system solves your users’ actual problem — a model can score well on public benchmarks and still underperform on your domain-specific task.

Should I use reference-based or reference-free evaluation? Use reference-based evaluation when you have a known correct answer — development-time regression testing and stress tests. Use reference-free evaluation for production monitoring, guardrails, and open-ended conversation where no single correct answer exists.