The Output Contract: A Pattern for Testable AI Systems

This is part 3 of the AI Design Patterns series:

Part 1: Why We Actually Need Design Patterns for AI
Part 2: The Semantic Orchestrator: Why Text-to-SQL Is the Wrong Answer
Part 3: The Output Contract: A Pattern for Testable AI Systems ← You are here

LLM-as-judge has become the default answer to "how do we know if this is working." Pipe the output into another model, ask whether it's good, treat the score as a signal.

It's a smell. And understanding why tells you what to build instead.

The question before the eval question

Before you ask "how do we evaluate this system," ask a harder question: does this system make claims, or does it make suggestions?

A creative writing assistant makes suggestions. There's nothing objectively correct or incorrect about whether a sentence is evocative. A meeting summarizer produces something useful or not — but "useful" is a matter of preference, not truth. These systems don't need rigorous evals. They need human judgment, and applying measurement frameworks to them creates false precision. You're not evaluating correctness; you're measuring taste.

A system that generates a historical report of the taxes you paid last year is different. There is an objective truth. The numbers either match the records or they don't. The dates are correct or they aren't. This system makes claims, and claims can be verified.

This distinction matters more than any framework: every production LLM system that makes claims must have at least one component of its output that is objectively verifiable by deterministic code. If your system has no such component — if every field is a matter of opinion, tone, or quality judgment — then you don't have an eval problem. You have a design problem. You've built a system that makes claims it structurally cannot be held to.

If you can't measure it, you can't know if it's right. And if you can't know if it's right, you shouldn't be shipping it as though it is.

What reaching for LLM-as-judge is actually admitting

When LLM-as-judge is your primary eval strategy, you've made an implicit admission: your outputs are so unstructured that the only thing capable of interpreting them is another probabilistic model. You've added a second stochastic layer on top of the first.

But teams don't reach for LLM-as-judge because LLMs are hard to evaluate. They reach for it because nobody committed to a definition of "correct" before shipping. The output is a text blob. The eval is a vibes check. LLM-as-judge provides cover — a number you can point to that doesn't require anyone to own a threshold or sign off on a specification.

It's comfortable. That's the problem. Comfort here means deferred decisions. And deferred decisions about what "good" means are technical debt you're accumulating in the most load-bearing part of your system.

That's a design problem, not an evaluation problem. And it has a design solution.

The output contract

The first move happens before you write a single eval: identify which fields in your output represent objective truth.

action: "refund" is either the right action or it isn't. amount: 50.00 either matches the original transaction or it doesn't. customer_id either belongs to the authenticated session or it doesn't. These fields make claims. They can be right or wrong in a way that code can verify.

explanation: "The charge appeared to be a duplicate based on the transaction history" is different. It may be accurate, helpful, well-phrased — but there's no ground truth to assert against. It's a suggestion, not a claim.

Your output contract separates these explicitly. The claim fields are your tier 1 anchors — the parts of the system that must be correct, verified by deterministic code, blocking on failure. The suggestion fields are candidates for fuzzy evaluation. This separation isn't just a testing convenience; it's a design statement about what your system is responsible for getting right.

The schema should represent the decision the system made, in terms specific enough to assert against. Not just valid JSON — that's table stakes.

The difference between these two outputs is everything:

"I've processed your refund of $50 for the duplicate charge on your account."

{
  "action": "refund",
  "amount": 50.00,
  "reason": "duplicate_charge",
  "customer_id": "cust_8821",
  "confidence": "high"
}

The first requires interpretation. The second can be tested with code. action is an enum — either it's a valid value or it isn't. amount is a number you can compare against the request. customer_id can be verified against the session. confidence is a signal you can route on.

Here's why this matters beyond testability: defining the contract creates the surface for a conversation that most teams avoid. What fields does this output need? What are the valid values for action? Who decides when confidence is "high" versus "medium"? These aren't technical questions. They're product and organizational questions — what does this system actually do, who owns the definitions, what does "correct" mean for each field?

The contract doesn't answer those questions automatically. It makes them impossible to defer. A schema with an action enum doesn't tell you when refund is the right answer — that's a specification, and it has to be written separately, owned by someone, and reviewed when the business logic changes. The contract is the structure. The specification is the meaning. You need both.

LLM-as-judge lets you ship without either. The output contract forces the structure into existence and makes the specification gap visible. That's the point.

The contract is the foundation. Everything downstream — what to test, how to test it, whether a failure blocks a deploy — flows from the contract design.

The assertion hierarchy

Once your outputs have structure, you evaluate them in tiers, cheapest and most reliable first.

Tier 1: Deterministic assertions

Hard checks on the claim fields — the parts of the output that are objectively true or false. No model required, no ambiguity, pass or fail.

action must be one of ["refund", "credit", "escalate", "dismiss"]
amount must be positive and not exceed the original transaction value
customer_id must match the authenticated session
Required fields must be present

These aren't just quality checks. They're the proof of correctness for the system. If a deterministic assertion fails, the system made a wrong claim — an action that doesn't exist, a negative dollar amount, a hallucinated customer ID. These block a deploy the same way a failing unit test does, and for the same reason: correctness is not optional.

Fast, cheap, run on every commit. This tier is also the one you can't skip. A system with no tier 1 assertions is a system with no verified claims — which means, per the earlier framing, a system that probably shouldn't be in production making decisions.

Tier 2: Semantic similarity

Some fields are genuinely fuzzy. The explanation text for why a refund was granted. The classification of a support ticket's intent. The summary of a long document. These can't be reduced to enum checks.

For these, vectorize both the expected output and the actual output and measure cosine similarity. Close vectors mean the model did roughly the right kind of thing.

Critical caveat, and it matters: cosine similarity measures semantic proximity, not factual correctness. "Your refund of $50 has been processed" and "Your refund of $500 has been processed" will score high similarity — same structure, same intent, same phrasing. But one of them costs you $450. Never use cosine similarity on fields where correctness is binary. Use it only where gradient quality is what you're actually measuring: tone, intent classification, explanation fluency.

Threshold calibration is required. Set a minimum similarity score per field; a drop below it fails the build. Review those thresholds periodically — model behavior drifts, and a threshold calibrated in March may no longer reflect acceptable quality in September.

Tier 3: LLM-as-judge

For genuinely subjective criteria — helpfulness, nuanced reasoning quality, whether the response hedged appropriately on an uncertain question — LLM-as-judge is sometimes the right tool. The key word is sometimes.

This tier is the noisiest, most expensive, and hardest to calibrate. Use it only where tiers 1 and 2 genuinely can't reach. Never use it where a deterministic check is possible. Never gate a deploy on it.

What it's good for: trend tracking. Run it on a schedule, watch scores over time, treat a sustained drop as a signal to investigate. A single bad score is noise. A week of declining scores is a problem.

The eval pyramid

This mirrors the testing pyramid deliberately.

         /\
        /  \  LLM-as-judge
       /----\  (few, slow, non-blocking — track trends)
      /      \
     /--------\  Semantic similarity
    /          \  (some, moderate cost — threshold gates)
   /------------\
  /              \  Deterministic assertions
 /________________\  (many, fast, cheap — blocking)

Most of your evals should be at the base: deterministic, fast, cheap, run constantly. A smaller number use semantic similarity with calibrated thresholds. A handful use LLM-as-judge for subjective quality you can't otherwise capture.

The output contract design determines which tier each field belongs to. An action enum is tier 1 by definition. A free-text explanation is tier 2 or 3 depending on precision requirements. This isn't a judgment call you make at eval time — it's a consequence of how you designed the contract.

CI integration: two modes

Blocking gates apply to tiers 1 and 2. Deterministic failures are hard blocks — the build doesn't ship. Semantic similarity failures are threshold blocks — drop below the floor, the build fails. Both are fast enough to run on every commit.

Trend lines apply to tier 3. LLM-as-judge scores go into a dashboard, not a gate. You're not asking "did this commit break something?" — you're asking "is quality drifting over time?" Different questions, different tooling. A monitoring dashboard with alerting on sustained decline is right. A CI check that fails 30% of the time for reasons unrelated to your commit is not.

The CI structure follows directly from the contract: which fields get hard checks, which get thresholds, which get trend monitoring. It's not trivial to implement, but it's unambiguous to design once the contract exists.

The failure mode of the pattern itself

There's a failure mode worth naming honestly: the contract can be wrong.

An action enum that turns out to be the wrong decomposition of user intent. A similarity threshold that was calibrated on clean training data and never updated as real user behavior diverged. A field you didn't include in the contract that turns out to be where the most interesting failures happen.

The contract is only as good as your understanding of the domain at the time you designed it. In the early weeks of a new system, that understanding is limited. Tier 1 assertions will pass green while the system fails in ways the contract never anticipated — because the contract was too narrow, or framed the decision incorrectly, or missed an edge case that only real usage surfaces.

The mitigation isn't to delay shipping until the contract is perfect. It's to treat the contract as a living spec. When your LLM does something wrong in production, you don't just fix the model — you add an assertion that would have caught it. The eval suite grows alongside the system, the same way a test suite does. The discipline is the same: no regression without a test, no LLM failure without a new eval.

That's what "first-class system" actually means. Not a fancy framework. A commitment to maintaining the specification as the system evolves.

What it costs to defer

LLM-as-judge is comfortable because it lets you avoid the hard questions. Nobody has to commit to a definition. Nobody owns the threshold. The judge says it's fine, so it's fine.

The output contract pattern is uncomfortable for exactly the same reason. It forces every "what does correct mean?" conversation into the open, before you ship, when you still have time to act on the answers. It requires someone to own the spec. It requires product and engineering to agree on what the system is actually supposed to do.

That's not a technical cost. It's an organizational one. But it's also the only way to build a system you can reason about when something goes wrong at 2am.

The alternative isn't "ship faster." It's ship now, and pay later — in the form of a probabilistic eval layer you can never fully trust, running on top of a system whose correct behavior was never precisely defined.

The deeper issue is that LLM-as-judge as a primary strategy mistakes preference for correctness. A score from a judge model tells you whether the output seemed good to another LLM, not whether the system got the right answer. For subjective products — creative tools, tone rewriters, style assistants — that distinction doesn't matter. For systems making claims about the real world, it's the only distinction that matters.

Build systems that can be measured. If you can't identify a single claim your system makes that can be verified by code, that's the design problem to solve first.

When to use it (and when not to)

The Output Contract pattern is right for:

Production systems where the LLM makes decisions with real consequences — financial, medical, legal, operational
Any system where a wrong output is worse than no output
Teams that need to ship LLM features with CI integration and regression protection

It's the wrong choice for:

Purely subjective products — creative writing assistants, style rewriters, brainstorming tools. There's no objective correctness to assert against, and applying the pattern creates false precision without adding reliability
Prototypes and early explorations where the right output shape isn't known yet. Define the contract once you understand what the system should do, not before
Systems where every output field is genuinely fuzzy. If you can't identify a single claim field for tier 1, the pattern is telling you something about your system design, not failing you as a tool

The boundary between "claims" and "suggestions" isn't always obvious. A customer support summarizer might feel subjective, but "the ticket includes a refund request" is a claim. Look for the decision the system is making on the user's behalf — that's where the claim fields live.

Starting from zero: building the initial eval dataset

The pattern describes how to maintain evals over time. The harder question is where to start.

For tier 1 assertions, the initial dataset is your specification: the valid enum values, the field constraints, the required fields. Write them before you ship, from the same conversation that produces the output contract. If you can't write them, you haven't finished defining what the system should do.

For tier 2 semantic similarity, you need calibration pairs — expected inputs matched to expected outputs. Three sources work in practice: hand-curated examples from domain experts (expensive, high quality), synthetic examples generated by the model itself and reviewed by a human (faster, requires judgment), and real production traffic once the system is live (best signal, available later). Start with 20–50 curated examples per field. That's enough to set an initial threshold. Revisit the threshold after 30 days of production data.

For tier 3 LLM-as-judge, you need rubrics before you need data. Define what "helpful" means for your context before you ask a judge model to evaluate it. A rubric forces the same conversation the output contract forces for tier 1: what does correct look like? Without a rubric, LLM-as-judge scores are measuring the judge's priors, not your system's quality.

The initialization investment is real but bounded. A team that spends two days defining the output contract, writing 30 curated tier 1 examples, and drafting tier 2/3 rubrics before launch is not doing extra work — they're doing the specification work that would otherwise happen reactively, under pressure, after something breaks in production.

How this pattern fits with others

The eval hierarchy answers "how do you know if a single output is correct." It doesn't answer questions about systems that accumulate state — multi-turn conversations, orchestrators that call multiple tools in sequence, memory that persists across sessions. In those contexts, the output contract still applies at each step, but the interesting failures often emerge from the interaction between steps, not from any single output in isolation.

Other patterns in this series address that layer. The Semantic Orchestrator pattern, for instance, is a natural source of claim-bearing structured outputs — each tool call the orchestrator makes is a candidate for a tier 1 assertion. When building both, apply the output contract at the tool level as well as the final response level. The stale tool definition failure mode in that pattern and the wrong output contract failure mode in this one are the same underlying problem: a specification that was correct when written and drifted without anyone noticing.

	Ad hoc evals	Output Contract pattern
Reliability	LLM-as-judge all the way down	Deterministic where possible
CI integration	Manual / judgment call	Automated, tiered, blocking where appropriate
Maintenance	Evals drift from system behavior	Output contract is a living spec
Setup cost	Low	Medium — contract design is the real work
Coverage	Hard to know what you're missing	Hierarchy makes gaps explicit