The assertion that stopped working

Here is a test. It is not a sophisticated test, and that is the point.

response = assistant.ask("Explain the cancellation policy")

assert response == expected_response

For most of the history of test automation, this shape has been enough to carry an enormous amount of engineering weight. It is cheap to write, cheap to run, and unambiguous when it fails. It gives a binary answer, a diff, and a starting point for debugging. Whole regression suites are built out of little else.

Now run it against a system with a generative component. Here are four responses the system might produce, on four consecutive runs, with the same input and the same configuration.

(A) You can cancel any time before the renewal date. Cancellations after
    renewal are not refundable.

(B) Cancellation is available at any point in the billing cycle. If you
    cancel before your renewal date, you will not be charged for the next
    period. Once a renewal has been processed, that charge is final.

(C) Two things matter here: the renewal date, and whether a charge has
    already gone through. Before renewal, cancel freely. After renewal,
    the current period is non-refundable.

(D) Sie können Ihr Abonnement jederzeit vor dem Verlängerungsdatum
    kündigen. Nach der Verlängerung ist die laufende Periode nicht
    erstattungsfähig.

All four are, for the purposes of a customer support product, acceptable. They encode the same two facts: cancellation is available at any time, and the current period becomes non-refundable once renewal has been processed. They differ in length, in ordering, in register, and — in the fourth case — in language. A reviewer looking at any one of them in isolation would sign it off.

The equality assertion passes for at most one of them, and only by accident. Which one it passes for depends on which output happened to be captured on the day someone wrote the test.

There is a tempting first reaction, which is to say that the test was badly written and that string equality was always a crude oracle. That reaction is only half right. The equality assertion was not crude; it was precise, and precision is what made it useful. What has changed is that the precision is now aimed at the wrong property. The test is asserting on the surface form of the answer, and the surface form is no longer part of the specification. Nobody ever promised that the assistant would phrase the cancellation policy in exactly 19 words in that particular order. That was an artifact of one execution that got frozen into a file.

So delete the assertion. That is the second tempting reaction, and it is worse. A test function that calls the system and asserts nothing is not a weak test; it is a smoke check dressed as a test. It will detect exceptions, timeouts, and total service failure. It will not detect the response that confidently states a 45-day refund window when the policy says 30. It will not detect the response that quotes another customer's account balance. It will not detect the agent that issued a refund when it was only asked to explain how refunds work. Those are exactly the failures that matter, and they all produce syntactically fine, superficially plausible text.

This is the actual problem, and it is sharper than "AI outputs vary":

Multiple outputs can be simultaneously valid, while some outputs that look very similar to the valid ones are seriously wrong.

Both halves are load-bearing. If only the first half were true — if variation were always benign — we could simply stop asserting on generated text and test everything around it. If only the second half were true — if there were one right answer and the system sometimes missed it — we could keep equality and just fix the model. Because both are true at once, the test needs to become discriminating in a way that string comparison cannot be. It has to separate the differences that are noise from the differences that are defects, and it has to do that automatically, thousands of times, in a suite that runs without a human reading the output.

Which leads to the question this article is built around:

If twenty different outputs could all be correct, what exactly should an automated test assert?

Not "how do we evaluate LLMs." Not "which platform should we buy." The narrower and more useful engineering question: what goes on the right-hand side of the assertion, once the left-hand side stopped being a fixed string?

The oracle, and the problem with it

The formal name for the thing that broke is the test oracle. In testing literature, the oracle is whatever mechanism decides whether observed behavior is correct. Barr, Harman, McMinn, Shahbaz, and Yoo's survey of the field defines the challenge plainly: given an input, distinguishing correct behavior from potentially incorrect behavior is the test oracle problem, and automating that distinction is what unlocks broader test automation in the first place — without it, a human has to look at every result (Barr et al., IEEE TSE, 2015).

That framing is more than three decades old and predates every generative model in production today. It is worth holding onto, because it reframes what we are dealing with. We are not facing a new category of problem invented by language models. We are facing an old, well-studied problem that generative systems have pushed into the middle of ordinary application development.

Most working engineers use oracles constantly without naming them. The common ones:

Notice that these are not equally strong, and they were never equally strong. The snapshot oracle in particular has always been a compromise: it does not encode a requirement, it encodes what happened last time somebody looked. Snapshot tests were tolerable when output was stable, because a diff meant "something changed, go look." They collapse when output is legitimately different on every run, because then every run is a diff and every diff is meaningless.

The generative case strains the oracle in several specific ways at once:

• The output space is enormous. For a numeric function, the set of plausible-but-wrong outputs is usually small and structured. For free text, it is effectively unbounded, and wrong answers sit arbitrarily close to right ones in surface form.
• Correctness is partly semantic. Whether an answer is right depends on meaning, not tokens. Meaning is not directly observable by the test harness.
• The interface is natural language in both directions. Inputs vary too. There is no canonical form of "the user asked about cancellations."
• The system takes actions. Once a model can call tools, the observable behavior includes side effects, and side effects can be irreversible.
• Behavior depends on context assembly. The same user question with a different retrieval result is, functionally, a different input — even though the test wrote only the question.
• The available automated oracles are themselves uncertain. If you evaluate meaning with a second model, you have introduced a second stochastic system into the test. More on this later, because it deserves it.

Two slogans are worth writing on the wall before going further, because nearly every bad decision in this space comes from ignoring one of them:

different  !=  incorrect
similar    !=  correct

The first kills equality assertions and snapshot diffs. The second kills naive similarity scoring, which is the most common overcorrection. A response that scores 0.94 cosine similarity against your reference answer can still contain the one number that makes it wrong. We will come back to this with a concrete example, because it determines the shape of a lot of the design that follows.

Nondeterminism did not arrive with language models

There is a version of this discussion that treats variable output as a novelty. It is not, and pretending otherwise makes the engineering worse, because it discards decades of applicable technique.

Software has been producing legitimately different results on identical inputs for as long as we have had concurrency. Thread interleaving, lock acquisition order, and scheduler decisions produce different execution paths. Distributed systems return different results depending on which replica answered and how far it had caught up. Hash-ordered iteration changes output ordering between runs and between language versions. Randomized algorithms — Monte Carlo methods, randomized quicksort, simulated annealing, load balancers, A/B assignment — are nondeterministic by construction. Search and ranking systems have never had a single correct ordering; relevance is a judgment, and relevance judgments disagree. Recommender systems return different item sets to the same user on the same day. Floating-point reduction order changes numeric results on GPUs. Timestamps, UUIDs, and auto-increment IDs have been forcing engineers to write tolerant assertions since forever.

All of that produced real testing practice: property-based testing, metamorphic relations, statistical acceptance criteria, deterministic seeding, dependency injection, record/replay harnesses, tolerance-based numeric comparison, idempotency checks. Almost everything in this article has a pre-LLM ancestor. That is good news. It means the problem is tractable and the techniques are proven; what is new is the combination and the intensity.

What is genuinely harder about generative systems, stated precisely and without inflation:

1. The valid output set is large, unenumerable, and semantically defined. With randomized quicksort you still assert sorted(output). The property is crisp. With a generated explanation, the property is "conveys these facts, invents none, contradicts nothing" — true, checkable in principle, but not with a one-line predicate.
2. Correctness depends on context that the test may not fully control. Retrieval, tool responses, conversation history, and system prompts all participate. Changing any of them changes the effective input.
3. Behavior emerges from a component nobody specified line by line. You cannot read the model's implementation to determine which paths exist.
4. Small input perturbations can produce large behavioral changes. Prompt wording, document ordering, and irrelevant context can shift outcomes in ways that are hard to predict from first principles.
5. The system is upgraded underneath you. A model version change can alter phrasing, verbosity, tool-selection style, and reasoning depth — all at once, without any change in your repository.

That last point has an underappreciated consequence for test design, and we will return to it under the heading of oracle brittleness.

Say what kind of variance you actually mean

Precision in vocabulary pays off in triage, so it is worth fixing terms:

• Nondeterministic — the same input can yield different outputs on different executions. This is a property of the system as observed.
• Stochastic — the variation is governed by a probability distribution the system samples from. Sampling with a temperature above zero is stochastic.
• Probabilistic — the system reasons in terms of probabilities, or outputs them. A classifier emitting a confidence is probabilistic; it may still be fully deterministic.
• Variable — the output differs, without a claim about the mechanism.
• Flaky — a test produces inconsistent verdicts without a corresponding change in the behavior under test. Flakiness is a property of a test, not of a product. Calling a genuinely unstable product "flaky" is how instability gets normalized.

And a system can appear nondeterministic for reasons that have nothing to do with sampling:

This table is doing real work. When a test fails intermittently, the first job is to determine which row you are in. Four of these eight rows are ordinary engineering problems with ordinary solutions, and teams routinely misattribute them to "the model is nondeterministic" and give up.

Even the sampling row is less mystical than it is usually treated. A widely discussed 2025 analysis argued that the dominant cause of run-to-run variation on hosted endpoints is not sampling at all but the fact that inference kernels are not batch-invariant: the numerical result for your request depends on the batch it was grouped into, and batch composition varies with server load. Make the kernels batch-invariant and identical requests produce identical outputs, at a throughput cost (He and Thinking Machines Lab, 2025; the accompanying batch_invariant_ops implementation demonstrates it under vLLM). Most teams will not be operating their own inference stack, so this is not directly actionable for them — but it changes the mental model. Some of what looks like irreducible model randomness is infrastructure behavior, and infrastructure behavior can be engineered.

That distinction matters for one practical reason. If variation is a property of the universe, you can only measure it. If variation is a property of a configuration, you can sometimes remove it and get a much cheaper test.

Figure 1 — "Sources of observed variance." What it shows: A request flowing left to right through user input → context assembly (retrieval, history, system prompt) → model inference → tool calls → response formatting, with each stage annotated by the kind of variance it can introduce and whether that variance is controllable in a test environment (green = controllable, amber = partially, red = inherent). Caption: "Not all variation is model variation. Most of the boxes in this diagram can be frozen." Placement: Immediately after the source table above.

Oracle 1: keep the exact assertions you still have

The most common failure of judgment in this area is enthusiasm. A team discovers that free-text output cannot be compared with ==, concludes that the system is "nondeterministic," and starts routing everything through a semantic evaluator — including the parts that were never nondeterministic in the first place.

This is expensive, slow, and less accurate. Consider:

# Don't
verdict = judge("Does this look like valid JSON with the right fields?")
assert verdict.score > 0.8

versus:

# Do
payload = json.loads(raw)
jsonschema.validate(payload, RESPONSE_SCHEMA)

The second is not merely cheaper. It is correct, in the strong sense: it will never say a malformed payload is fine because it read plausibly, and it will never fail a valid payload because the judge was in a bad mood. Schema validity is a decidable property. Handing a decidable property to a probabilistic evaluator is a category error.

The same goes for identity checks. If the test knows that the request concerned customer C-1842, then this:

assert trace.tool_calls[0].arguments["customer_id"] == "C-1842"

is strictly better than asking a model whether the assistant looked up the right account. There is no interpretation involved. There is nothing to calibrate. It runs in microseconds.

An AI-enabled application is not uniformly stochastic. It is a mostly ordinary program with a stochastic component somewhere in the middle, and it retains a large surface of hard, checkable facts:

• HTTP status codes and error taxonomies
• JSON parseability and schema conformance
• Presence and type of required fields
• Enumerated values drawn from a closed set
• Number of returned objects, page sizes, cursor behavior
• Identifiers: customer, order, document, tenant
• Authorization outcomes and permission boundaries
• Which tools were invoked, with which arguments
• Database rows written, updated, or (critically) not written
• Idempotency keys and duplicate suppression
• Arithmetic the application performs itself
• Timestamps falling inside a defensible window
• Latency and token budgets
• Emitted events, webhooks, audit log entries

Every one of these is an oracle you already know how to write, and each has the four properties that make automated tests worth having:

• Low noise. A deterministic check does not produce different verdicts on identical inputs.
• Reproducibility. A failure can be re-run and re-observed.
• Cheap execution. Thousands of these can run per commit.
• Sharp diagnostics. expected "C-1842", got "C-1841" needs no interpretation.

So the first principle of testing a nondeterministic system is defensive rather than innovative:

Preserve determinism wherever the product still gives it to you, and spend semantic oracles only where nothing cheaper will do.

There is a corollary that shapes architecture, which we will develop later: if too little of your application is deterministically checkable, that is often a design problem rather than a testing problem.

Oracle 2: invariants — what must hold no matter what changes

Once the deterministic surface is exhausted, the next question is not "what should the output be" but "what must be true of any acceptable output." That is an invariant.

An invariant is a statement about the system's behavior that holds across the entire space of valid outputs. It is not a description of one answer; it is a constraint on all of them. This is the single most important conceptual move in testing nondeterministic software, because it converts an unanswerable question ("is this the right text?") into a set of answerable ones ("does this text violate any of the things that must never be violated?").

Some invariants for a customer support assistant:

I1  Any policy duration stated in the answer must appear in the retrieved
    policy documents. The assistant may not originate a number.

I2  If the customer's plan carries a cancellation fee, the answer must
    surface it. Silence about a cost is a defect, not a style choice.

I3  Any account-specific claim (balance, renewal date, plan tier) must be
    traceable to data returned by an account tool during this request.

I4  No content from another tenant's records may appear in the response,
    under any phrasing.

I5  No irreversible action may be executed without an explicit user
    confirmation turn preceding it.

I6  If the retrieved context does not support an answer, the assistant
    must say so rather than produce one.

Read those again and notice something: none of them mention wording. All six survive translation into German. All six survive an arbitrary rewrite of the response in a different register. They are properties of what the system asserted and did, not of how it phrased it. That is what makes them stable oracles.

Invariants for a summarization feature look different but have the same shape:

S1  Every named entity in the summary appears in the source document.
S2  Every monetary amount in the summary appears in the source, with the
    same currency and the same magnitude.
S3  Every date in the summary is present in, or derivable from, the source.
S4  The summary introduces no event that the source does not describe.
S5  Negations are preserved: if the source says a payment was declined,
    the summary must not say it was processed.

S5 is worth pausing on, because it is the kind of failure that similarity metrics are structurally bad at catching. "The payment was processed" and "the payment was not processed" are lexically almost identical and semantically opposite.

Invariants for a structured extraction pipeline are the most tractable of all, because the output is already machine-readable:

E1  line_item_total == sum(line_items[].amount)
E2  invoice_total == line_item_total + tax - discounts   (± rounding tolerance)
E3  start_date <= end_date
E4  currency ∈ SUPPORTED_CURRENCIES
E5  every extracted value appears verbatim, or in a normalized form, in
    the source document text
E6  required fields present; unknown fields absent

The four kinds of invariant you will actually write

It helps to classify these, because each class has a different implementation cost and a different failure mode.

Deterministic invariants are computable directly from the output and the inputs. sum(line_items) == total is one. So is "the response contains no string matching a credit card pattern." These are cheap, exact, and should be your first choice. A surprising number of apparently semantic requirements collapse into deterministic checks once you look at them: "must not invent a refund period" becomes "every duration-like token in the response must appear in the retrieved context," which is a regex plus a set membership test.

Relational invariants compare the output against another artifact: the source documents, the tool results, the previous turn, the database state before and after. These are where groundedness lives. "Every account fact in the response is present in the tool result" is relational, and it is checkable without a judge if you have structured tool results and are willing to do entity-level comparison.

Semantic invariants require interpreting meaning and cannot be reduced to string operations. "The answer does not contradict the source" is semantic. "The answer addresses the question that was asked" is semantic. These need a semantic oracle, with all the caveats that implies.

Domain invariants encode business or regulatory rules that no general-purpose evaluator knows about. "Refunds above €500 require a supervisor approval step." "Health-related queries must include the standard referral text." "Customers in jurisdiction X must be shown the statutory cancellation window rather than the contractual one." These are the highest-value tests in most systems, because they are the ones that map directly onto real-world consequences — and no off-the-shelf evaluator will ever produce them for you.

Invariants change what a test suite is for

This shift deserves to be stated explicitly, because it reorganizes how you think about coverage. In a conventional suite, a test case is an example: this input produces this output. In a suite built on invariants, a test case is closer to a claim about the space of behaviors, checked against one or more samples drawn from that space.

The practical consequence is that a single invariant can be evaluated across your entire test corpus rather than being attached to a single case. "No response ever states a policy duration absent from context" is not one test; it is a predicate you run over every generated response in the suite, including the ones written to check something else entirely. That is enormously more coverage per unit of effort than writing one expectation per input, and it is the closest thing to a free lunch in this domain.

You end up with a suite that looks less like a list of expected outputs and more like a specification with sampling attached: a body of properties, plus a corpus of inputs, plus a policy for how many times to sample each.

Figure 2 — "Invariants as a boundary around a variable output." What it shows: A shaded region labelled "acceptable outputs" containing several scattered, visibly different response bubbles (concise, verbose, reordered, translated). The region's boundary is drawn from labelled constraint lines: "no unsupported numbers," "required fee disclosed," "no cross-tenant data," "no action without confirmation." Outside the boundary, two response bubbles that are lexically close to the inside ones but cross a line. Caption: "Correctness is the region, not the point. Two outputs can be nearly identical in wording and land on opposite sides of a boundary." Placement: At the end of the invariants section.

Properties instead of examples

Everything above is a rediscovery, in a new setting, of property-based testing — an approach that has been in production use since QuickCheck introduced it for Haskell in 2000 and that reached mainstream practice through tools like Hypothesis for Python, fast-check for JavaScript, and jqwik for the JVM.

The core idea of property-based testing is that instead of writing f(2, 3) == 5, you state a property that should hold for all inputs — f(a, b) == f(b, a), or decode(encode(x)) == x — and let a generator produce many inputs, including adversarial ones, looking for a counterexample. When it finds one, it shrinks it to the smallest failing case.

The intellectual overlap with what we are doing is exact: both replace an enumerated expectation with a universally quantified constraint. The differences matter too, and glossing over them produces disappointment.

What transfers well:

• The habit of asking "what must be true of all valid outputs?" rather than "what is the output?"
• Input generation. If you have a property like "no unsupported numbers appear in the answer," you can throw a hundred paraphrased questions at it rather than one.
• Shrinking as a debugging discipline: when a property fails, reduce the input until you have the minimal trigger. This works on prompts and contexts, not just data structures.
• The separation between the generator (what inputs to try) and the oracle (what must hold).

What does not transfer:

• Classical property-based testing assumes the property is cheaply and exactly decidable. Many of ours are not. sorted(xs) is a function; "does not contradict the source" is a judgment.
• Shrinking assumes a well-ordered input space with a meaningful notion of "smaller." Natural language has no canonical shrinking order — though you can approximate it by removing context documents, shortening the question, or dropping conversation turns.
• Each execution is expensive. A property-based suite that runs a thousand generated cases per property is normal in classical testing and is a budget conversation when every case is a model call.
• Failures may be probabilistic. A classical counterexample reproduces; a generated counterexample may reproduce two times in five.

That last point is important enough that it gets its own section later. For now, the takeaway is a reframing:

# The example-based oracle
assert output == "The refund period is 30 days."

# The property-based oracle
Given: the retrieved policy states a 30-day refund period.
Require: the response states 30 days, or states no period at all;
         it must never state a different period.

The second version survives paraphrase, translation, reordering, verbosity changes, and model upgrades. It also catches a class of failure the first version misses entirely: the first assertion passes only on one exact sentence, but it also fails silently in the sense that it tells you nothing about the hundred other phrasings the system might produce in production. The property covers all of them.

Note the shape of the property carefully. It is asymmetric: it permits omission and forbids contradiction. That is a deliberate design decision reflecting the fact that a missing fact and a wrong fact usually carry very different severities. Whether you want that asymmetry depends on your domain — in a compliance disclosure, omission is the defect — but it should be a decision, not an accident.

Fuzzing, and what it means here

Fuzzing contributes a different instinct: throw malformed, hostile, and unexpected inputs at the system and check that it fails safely rather than catastrophically. Translated into this setting, that means building a corpus of inputs designed to break the rules rather than the parser:

• Requests that ask for another customer's data using plausible framing
• Requests that instruct the assistant to ignore its instructions
• Questions whose answers are not in the knowledge base at all
• Contradictory context documents
• Empty retrieval results
• Tool calls that time out or return errors
• Extremely long inputs that push context limits
• Inputs in languages the product does not officially support

The oracle for these is almost always an invariant rather than an expected output: no matter what this input does to the response, the system must not disclose, must not act, must not fabricate, must degrade gracefully. Fuzz-style inputs plus invariant oracles is one of the highest-yield combinations available, precisely because you do not need to know what the right answer is to know that a particular answer is wrong.

Metamorphic relations: testing without knowing the answer

Metamorphic testing is the technique that most directly addresses the oracle problem, and it deserves more space than it usually gets in discussions of AI testing.

The idea, introduced in the late 1990s and surveyed comprehensively by Chen and colleagues in ACM Computing Surveys, is this: when you cannot determine whether a single output is correct, you may still know how outputs should relate to each other when the input is transformed in a controlled way. The relation between inputs and the corresponding relation between outputs is a metamorphic relation, and violating it is evidence of a fault even though you never knew the right answer for any individual case.

The canonical illustration is a search engine. You cannot say what the correct result set is for a query. But you know that adding a restrictive filter should not increase the number of results, and that a query and its synonym should produce substantially overlapping results. Both are checkable without ground truth. The technique has been applied to compilers, machine translation, autonomous driving perception, web security, and bioinformatics — domains united by having no practical way to write down the expected output.

For systems built on generative components, metamorphic relations are unusually productive, because the inputs are easy to transform in meaning-preserving ways and the expected invariance is easy to state.

A working set of relations

Irrelevant-context insertion. Add material to the context that has nothing to do with the question — an unrelated policy page, a boilerplate footer, a different product's FAQ.

Transformation: context' = context + irrelevant_document
Relation:       the set of factual claims in the answer must be unchanged.

Violations here are informative. If adding an unrelated document changes which refund period the assistant quotes, the system is sensitive to context in a way that will bite in production, where retrieval quality varies.

Context reordering. Shuffle the retrieved documents.

Transformation: context' = permute(context)
Relation:       supported factual claims must be identical; citations may
                point to any document that genuinely supports the claim.

This catches position sensitivity — a well-documented behavior in which models weight material differently depending on where it sits in the context window. If your answer changes when document 5 becomes document 1, retrieval ranking has become part of your correctness surface, and you should know that.

Paraphrase invariance. Rewrite the user's request while preserving intent.

Transformation: "How do I cancel?" → "I want to end my subscription"
                                   → "cancel plz"
                                   → "What's the process for terminating
                                      my account?"
Relation:       the classified intent, the tools invoked, and the policy
                facts stated must be equivalent.

Note that this relation is about outcome, not text. The responses will differ in tone, and that is fine.

Capability removal. Take away a tool the agent had.

Transformation: tools' = tools \ {issue_refund}
Relation:       issue_refund must not appear in the trace; the agent must
                either accomplish the task another way or explain that it
                cannot.

This one is close to a safety property and is cheap to check, since it is a trace assertion.

Evidence removal. Delete the document containing the fact the answer depends on.

Transformation: context' = context \ {doc_containing_refund_period}
Relation:       the response must not state a refund period with confidence.

This is the single most useful metamorphic relation for retrieval-augmented systems, because it directly probes whether the answer was grounded in the supplied evidence or produced from parametric memory. An answer that stays identical after you remove its only evidence was never using the evidence. That is a real defect with a real production consequence — it means the system will keep answering confidently after your knowledge base is updated.

Evidence mutation. Change the fact in the source rather than removing it.

Transformation: policy document says 30 days → says 14 days
Relation:       the stated period in the response must change to 14, or
                the response must decline to state one.

Stronger than removal, and it catches the case where the model quietly prefers its own prior over your data.

Locale change. Request the answer in another language, with identical source facts.

Relation: numbers, dates, monetary amounts, and entity names must be
          identical after normalization; the set of asserted claims must
          be equivalent.

Authorization change. Run the same request as a user with fewer permissions.

Relation: the response must be a subset of the higher-privilege response
          in terms of disclosed facts, and must never contain a fact the
          user is not entitled to.

This is a metamorphic relation and a security test, and it is far easier to automate than trying to enumerate what a restricted user should see.

Conversation-order change. Provide the same information across turns in a different order.

Relation: the final task outcome must be the same.

When metamorphic relations beat fixed expectations

Three situations, specifically.

First, when you genuinely do not know the right answer — open-ended questions, summaries of long documents, recommendations. A relation sidesteps the need.

Second, when the answer is known but the phrasing is not, and you want to test robustness rather than a single case. One paraphrase relation applied over fifty inputs tests more than fifty golden answers do, because it tests consistency, which is a property no single golden answer can express.

Third, when you are testing for the reason an answer is right rather than the answer itself. The evidence-removal relation is the clearest instance. A test asserting "30 days" in response passes whether the system read your document or memorized the number in pretraining. The metamorphic version distinguishes those cases. That distinction is the difference between a system that will stay correct when your policy changes and one that will not.

The cost is real: metamorphic tests require at least two executions, sometimes more, so they multiply your model spend. They are also easy to get wrong — a relation stated too strongly ("the answer must be identical") will fail constantly and teach the team to ignore it. Relations should be stated over extracted properties, not raw text.

Figure 3 — "Metamorphic relation, concretely." What it shows: Two parallel execution lanes. Top lane: original question + 5 context documents → response → extracted claims {period: 30d, fee: none, immediate: true}. Bottom lane: same question + the same documents minus the policy page → response → extracted claims. Between them, a comparison box stating the relation: "claims about refund period must not survive removal of the only supporting document." A red marker where the bottom lane still asserts 30 days. Caption: "The relation, not the answer, is the oracle. This test can fail even when both responses read perfectly well." Placement: After the list of relations.

Oracle 3: acceptable sets, forbidden sets, and the drift from lists to policies

Some outputs are not free text at all. They are choices from a finite set: which label, which tool, which route, which template. Here the naive assertion is nearly right and just needs widening.

# Too narrow: encodes one valid choice as the only valid choice
assert action == "search_database"

# Better: encodes the set of acceptable choices
assert action in {"search_database", "search_cached_index"}

This is a real improvement, and for small closed decisions it is often the whole answer. But a static whitelist has a predictable failure mode: it accumulates. Every time a new model or prompt produces a new acceptable behavior, someone adds another member to the set. Six months later the set contains eleven tools, which is to say it asserts nothing. The test still passes, and it has quietly stopped being a test.

Three refinements keep set-based assertions honest.

Assert the complement. Instead of listing what is allowed, list what is forbidden. Forbidden sets are typically smaller, more stable, and tied to consequences rather than implementation.

FORBIDDEN_WITHOUT_CONFIRMATION = {"issue_refund", "cancel_subscription",
                                  "delete_account", "charge_card"}

assert not (called_tools & FORBIDDEN_WITHOUT_CONFIRMATION)

A forbidden set rarely needs to grow because a model got better at phrasing. It grows when your product acquires a new destructive capability, which is exactly when a human should be reviewing it anyway.

Make the set conditional. The interesting rules are almost never unconditional. get_account may be acceptable, but only after authentication has succeeded. issue_refund may be acceptable, but only if a confirmation turn preceded it and the amount is under a threshold.

def allowed_tools(state):
    tools = {"search_kb", "search_customer"}
    if state.authenticated:
        tools |= {"get_account", "get_orders", "calculate_refund"}
    if state.authenticated and state.user_confirmed_action:
        tools |= {"issue_refund"}
    return tools

for call in trace.tool_calls:
    assert call.name in allowed_tools(state_before(call)), (
        f"{call.name} invoked in state {state_before(call)}"
    )

This is no longer a whitelist. It is a small policy evaluated against the execution, and it expresses something a reviewer would recognize as a requirement.

Distinguish "acceptable" from "preferred." Two tools may both be legal while one is clearly better. Collapsing that into a single boolean throws away information. A better structure records both: a hard assertion on legality, and a soft metric on preference, tracked over the corpus rather than gating a single test.

assert call.name in legal_tools          # hard: fails the test
metrics.record("preferred_tool_rate",
               call.name == preferred_tool)  # soft: tracked, not gating

The hard assertion protects correctness. The soft metric tells you when a prompt change made routing worse without breaking it — which is the kind of regression that otherwise goes unnoticed until users complain.

Oracle 4: ranges, tolerances, and the temptation to widen them

Numeric outputs invite the assertion assert value == expected, and in a system with any stochastic element that assertion will fail. The standard replacement is a range:

assert lower <= value <= upper

Which is correct, and also the most abused pattern in this entire discipline, because a range has a dial on it and the dial can always be turned until the test goes green.

The discipline that prevents this is to be explicit about which kind of tolerance a given bound represents. There are at least four, and they come from completely different places:

Engineering tolerance comes from representation and arithmetic. Floating-point accumulation, rounding to cents, unit conversion. If two paths compute a total differently, they may differ in the last decimal place. The bound is derived from the arithmetic, not chosen: abs(a - b) < 0.005 for currency rounded to cents is a statement about the domain, not a preference.

Measurement noise comes from the observation, not the system. Latency measured on a shared CI runner varies by a factor of three. Token counts vary with tokenizer versions. This tolerance should be set from observed distribution — and preferably measured with percentiles rather than a mean, because the tail is where the user experience lives.

Model variance is the run-to-run spread of the system's own output at fixed input. A confidence score that lands between 0.81 and 0.86 across twenty runs has an empirical spread; a bound of >= 0.75 is defensible if you measured that spread and set the bound outside it. A bound of >= 0.5 chosen because 0.75 failed once is not a tolerance, it is a surrender.

Business tolerance is what the product can actually accept. An extracted invoice total that is off by €0.03 may be operationally irrelevant or may be a reconciliation failure, depending entirely on what downstream systems do with it. This is the only one of the four that is a product decision, and it is the one most often decided by an engineer at 6pm on a Friday.

Conflating these is how tolerance creep happens. If a bound is widened, the commit should say which category it belongs to and what evidence supports the new value. A useful convention is to encode the justification in the test itself:

# Engineering tolerance: currency values are rounded to cents at two
# different points in the pipeline; max representable divergence is 0.01.
assert abs(extracted_total - computed_total) <= 0.01

# Model variance: measured spread over 30 runs on 2026-07-14 was
# [0.79, 0.88]; bound set below observed minimum with margin.
assert confidence >= 0.75

Two further cautions. First, a range assertion that has never failed is not necessarily a good range assertion; it may simply be enormous. Periodically check the observed distribution against the bound and tighten if there is a large gap. Second, some numeric outputs should not have a lower bound at all but a directional constraint: a re-ranked result list should not be worse than the unranked one; a summarization should not be longer than its source. Relative constraints are often more meaningful and far more stable than absolute thresholds.

Oracle 5: semantic assertions, and their limits

At some point the deterministic surface, the invariants, the sets, and the ranges run out, and a genuinely semantic question remains: did this answer actually answer the question, without contradicting the evidence?

A semantic assertion is an automated check on meaning. It is worth being precise about what it is not. It is not a similarity score. It is not "does this look like the reference." It is a decision procedure — implemented by embeddings, by a rule set, by an entailment model, by a judge model, or by some combination — that returns a verdict on a specific, named semantic property.

The properties that turn out to be worth asserting are narrower than "quality":

• Answer relevance. Does the response address the question that was asked, as opposed to an adjacent one? (A common real failure: the user asks how to cancel; the response explains the refund policy.)
• Groundedness / faithfulness. Is every substantive claim supported by the supplied context?
• Contradiction. Does the response assert anything that the context denies?
• Completeness against a requirement. Does the response contain the elements the policy says it must — the fee disclosure, the deadline, the escalation path?
• Refusal correctness. When the system should decline, does it? When it should not decline, does it avoid over-refusing? Both directions matter; a suite that only tests the first produces a product that refuses to answer normal questions.
• Intent equivalence to a reference. Does this answer accomplish what the reference answer accomplishes, regardless of wording?

Why embedding similarity is not enough

The most common first implementation is cosine similarity against a reference answer, thresholded. It is fast, cheap, and deterministic, which are genuine virtues. It is also insufficient for the case that matters most, and the reason is easy to demonstrate:

Reference: "The warranty lasts two years from the date of purchase."
Candidate: "The warranty lasts three years from the date of purchase."

These sentences share structure, vocabulary, topic, and syntax. Any general-purpose embedding model will score them as highly similar, because they are highly similar — as language. As statements about the world, one is true and one costs the company a year of unplanned warranty liability per affected customer.

The same failure appears with negation ("the payment was declined" vs. "the payment was processed"), with entity swaps ("your order" vs. "order 8814"), and with quantifier changes ("all plans include support" vs. "some plans include support"). Embedding similarity measures distance in a space built for topical relatedness. Operational correctness frequently turns on exactly the tokens that contribute least to that distance.

The correct conclusion is not that similarity is useless. It is that similarity is a coarse filter, not an oracle. It is reasonable to use it to detect a response that has wandered entirely off-topic, and unreasonable to use it as the check that protects a numeric fact. The numbers need their own assertion.

The techniques, and what each is good for

The pattern that works in practice is layered rather than exclusive. Deterministic checks catch what they can. Extraction-plus-comparison handles the enumerable facts. A judge handles what remains, on a smaller number of cases, with the judge's own verdict treated as evidence rather than as truth.

Assert the facts, not the sentence

The most practically useful move in this entire area is also the least glamorous: stop asserting on the response and start asserting on the claims the response makes.

Take a response:

Your subscription remains active until September 30. You can cancel now
and continue using the service until then. No additional renewal will
occur after cancellation.

There are dozens of acceptable ways to say this. There is exactly one set of things it asserts:

end_date: 2026-09-30
can_cancel_now: true
access_until_end_date: true
renews_after_cancel: false
early_termination_fee: null

Once the response has been reduced to that structure, the oracle problem is largely gone. You are back to comparing values, which is what test automation is good at:

claims = extract_claims(response, schema=SUBSCRIPTION_CLAIMS)

assert claims.end_date == account.current_period_end
assert claims.can_cancel_now is True
assert claims.access_until_end_date is True
assert claims.renews_after_cancel is False
assert claims.early_termination_fee == account.plan.termination_fee

Every one of those assertions is exact. None of them constrains phrasing. The response may be one sentence or five, in any language, in any order.

How to get the structure

Four approaches, in increasing order of flexibility and decreasing order of reliability:

Have the system emit it. By far the best option when the architecture permits. If the application already produces a structured decision object internally — the dates, amounts, and flags that the natural-language response is generated from — then test the structured object and treat the prose as presentation. Many teams discover during this exercise that their application does not have such an object, and that building one is the highest-value change they could make for testability. We will return to this.

Domain-specific parsers. For narrow output types, a purpose-built extractor is entirely feasible: date parsing, currency parsing, entity lookup against a known list. A parser that extracts every duration expression ("30 days", "one month", "a month", "thirty days") and normalizes it is a few hundred lines of code and is deterministic forever. This is unfashionable and extremely effective.

Named entity and claim extraction with conventional NLP. For entities, dates, and amounts, standard extraction libraries are mature and deterministic.

Model-assisted extraction. Ask a model to convert the response into the structured claim set, then assert deterministically on the structure. Flexible, handles arbitrary phrasing — and introduces uncertainty into the oracle.

The extractor is now part of your test infrastructure

That last option deserves a hard look, because it is the one teams reach for first and reason about least.

If a model extracts the claims, then a test failure has two possible causes: the system said something wrong, or the extractor read it wrong. You have not eliminated the oracle problem; you have moved it one layer down. Sometimes that is a good trade — the extraction task is narrower and more constrained than the generation task, so the error rate is lower — but it is a trade, and it needs to be managed:

• Constrain the extraction task severely. "Return the refund period in days as an integer, or null if none is stated" is a task with a small output space and a checkable result type. "Summarize the key claims" is not.
• Force structured output. Schema-constrained decoding turns a class of extraction errors into parse failures, which are visible, rather than silent misreads.
• Cross-check extraction against deterministic evidence. If the extractor reports end_date: 2026-09-30, verify that the string "September 30" or an equivalent appears in the response. If it does not, the extractor hallucinated, and the test should error rather than fail.
• Measure extractor accuracy on a labelled set. You cannot trust an oracle whose error rate you have never measured. This is the same discipline that applies to judges, discussed later.
• Prefer extraction over judgment. "What number did it state?" is a much easier question than "was this answer good?" Push as much of the semantic work as possible into the first form.

There is one more structural benefit to claim extraction that is easy to miss. When a test fails, the failure message can say which claim was wrong and what it should have been. That is a defect report. A similarity score below a threshold is not.

Golden answers without golden-sentence overfitting

Reference answers are genuinely useful, and the way they usually get used is genuinely harmful. Both things are true.

What references are good for:

• Anchoring a discussion. A concrete example of a correct answer is the fastest way to align a team on what "correct" means for a case.
• Deriving properties. The reference is where the required facts come from. Read it, extract the claims, and turn those into assertions. The reference has done its job at that point; it does not need to appear in the test.
• Pairwise comparison. "Is the new output at least as good as this known-good output?" is a reasonable judge task, and more robust than absolute scoring.
• Regression anchors for known-fixed defects. When a specific failure has been fixed, the corrected behavior is worth recording.

What references are bad for: being the thing the output is compared against, character by character or embedding by embedding. The moment matching the reference becomes the objective, several things go wrong at once. Verbose references make verbose outputs look better. Alternative valid reasoning paths score as failures. A reference written eighteen months ago encodes product data that has since changed, and the test now enforces a stale fact. A model upgrade that produces genuinely better answers registers as a regression because the answers moved further from the reference.

The distinction worth internalizing:

golden answer   =  one acceptable output, recorded
golden behavior =  the set of properties that made it acceptable

The second is what you want in the suite. The first is what you keep in a fixture file for humans to read.

Concretely, the workflow is: write the reference, extract from it the facts and behaviors that make it correct, encode those as assertions, and store the reference alongside the test as documentation rather than as an oracle. If it turns out that you cannot articulate what made the reference correct, that is a valuable signal in itself — it usually means the requirement is underspecified, and the test you were about to write would have encoded an accident.

Trace-based testing: the answer is not the whole behavior

For anything with tools, retrieval, or multiple steps, the final response is an incomplete record of what happened. A correct-looking answer can be produced by an incorrect execution, and the difference is invisible from the output.

Consider a typical agentic request:

user request
    ↓
context assembly / retrieval
    ↓
model decision
    ↓
tool call ──────────→ tool result
    ↓
model decision
    ↓
tool call ──────────→ tool result
    ↓
final response

Failures that this structure permits, all of which can produce an acceptable final message:

• The agent looked up the wrong customer, got data that happened to be plausible, and reported it.
• The agent called a paid enrichment API eleven times where one call was required.
• The agent executed a refund and then described it as a proposal.
• The agent received a tool error, ignored it, and answered from memory.
• The agent answered correctly but never consulted the account at all, having guessed from the conversation.
• The agent retrieved a document the user is not authorized to see, used it, and did not cite it.

None of these is detectable from the response text. All of them are trivially detectable from the trace.

Outcome assertions and trajectory assertions

Two distinct questions, and the distinction organizes a lot of agent testing:

Outcome assertions ask whether the task ended in the right state. Was the refund issued? Is the database in the expected condition? Did the user get an answer to the question they asked? These are usually the primary criterion, and they have the great virtue of being route-agnostic — the τ-bench benchmark for tool-using agents evaluates precisely this way, comparing the final database state against an annotated goal state so that dialogue variation does not affect the verdict (Yao et al., 2024).

Trajectory assertions ask whether the route taken was acceptable. Which tools, in what order, with what arguments, how many times, and under what preconditions.

Outcome assertions alone are insufficient in a specific and important set of circumstances:

• Irreversible actions. "The refund was issued" is a fine outcome — unless the agent issued it twice and then reversed one. The end state is right; the customer received two emails and the finance team has a reconciliation problem.
• Permissions. An agent that accessed data it should not have, and then did not use it, has still committed a violation. There is no trace of it in the outcome.
• Cost. Twelve API calls and one API call produce the same outcome and very different bills.
• Compliance. Many regulated processes require a specific ordering — verify identity before disclosing account data, obtain consent before processing. The requirement is about the trajectory.
• Error handling. Whether a tool failure was handled or swallowed is only visible mid-trace.

The brittleness trap

The obvious way to write a trajectory assertion is the wrong one:

# Brittle: encodes one observed path as the requirement
assert [c.name for c in trace.tool_calls] == [
    "search_customer", "get_account", "calculate_refund", "draft_email"
]

This test says: do it exactly the way it was done on the day I wrote this. It will fail when the model discovers a legitimate shortcut, when a caching layer makes one lookup unnecessary, when the order of two independent calls swaps, or when a model upgrade produces a more efficient plan. Every one of those failures is a false failure, and each one erodes the team's willingness to trust the suite.

The better form expresses the constraints that actually matter:

names = [c.name for c in trace.tool_calls]

# Precedence: identity must be established before account data is read
assert index_of(names, "search_customer") < index_of(names, "get_account")

# Correct subject throughout
assert all(c.arguments.get("customer_id") == "C-1842"
           for c in trace.tool_calls if "customer_id" in c.arguments)

# Budget
assert names.count("calculate_refund") <= 2

# Prohibition
assert "issue_refund" not in names or trace.had_user_confirmation

# Required outcome
assert trace.final_state.draft_created is True

Every assertion here corresponds to something a reviewer would defend in a design discussion. None of them requires a particular sequence merely because it was observed once. This is the difference between recording behavior and specifying it.

A useful test when reviewing a trajectory assertion: if the agent did this differently but achieved the same result, would I actually want the build to fail? If not, the assertion is encoding an implementation detail.

Figure 4 — "Outcome vs. trajectory." What it shows: Two agent execution graphs side by side, both terminating at the same final state (refund drafted, correct amount). Path A: search → get_account → calculate → draft. Path B: get_account (cached) → calculate → draft. Both marked green under "outcome oracle." Under "trajectory oracle," a third path C is shown that also reaches the same final state but calls issue_refund before the confirmation turn — marked red. Caption: "Two valid routes and one violation, all with an identical final answer. Only a trace oracle separates them." Placement: After the trajectory assertion example.

Composite assertions: correctness as an intersection

By this point the shape of a mature test has emerged, and it is not a single comparison. It is a conjunction of constraints of different kinds, each cheap or expensive in proportion to what it covers.

A single realistic support-assistant test might require, simultaneously:

response parses and conforms to the response schema
AND  the account referenced is the requesting customer's
AND  no tool outside the allowed set for this state was invoked
AND  no destructive tool was invoked without confirmation
AND  every duration, amount, and date stated appears in retrieved context
AND  the applicable cancellation fee is disclosed
AND  nothing in the response contradicts the knowledge base
AND  the response addresses the question that was asked
AND  latency p95 within budget and token use within budget

Within that intersection, the response may be phrased in any way at all. It may be two sentences or ten, formal or casual, in any supported language. The test does not care, and should not care, because none of those dimensions was ever a requirement.

In code, this reads as a pipeline of independent checks with independent evidence:

def test_cancellation_with_fee(case):
    result = run(case.input, env=case.env)

    # deterministic
    schema.validate(result.payload)
    assert result.payload["customer_id"] == case.customer_id
    assert_tools_within_policy(result.trace, case.state)
    assert_no_destructive_calls(result.trace, confirmed=False)

    # relational invariants
    claims = extract_claims(result.text)
    assert_all_facts_grounded(claims, result.retrieved_context)
    assert claims.cancellation_fee == case.expected_fee

    # semantic, applied last and only if the cheap checks passed
    verdict = judge.assess(
        question=case.input,
        answer=result.text,
        context=result.retrieved_context,
        criteria=["addresses_question", "no_contradiction"],
    )
    assert verdict.addresses_question, verdict.rationale
    assert verdict.no_contradiction, verdict.rationale

    # budgets
    assert result.latency_ms < case.budget_ms
    assert result.tokens < case.budget_tokens

Two design notes about this ordering. First, the cheap deterministic checks run first, so that a schema violation fails fast without spending a judge call. Second, each assertion carries its own evidence, so a failure names the property that broke rather than reporting an aggregate score. Both of those choices matter more than they look; we will come back to failure evidence specifically.

Figure 5 — "Correctness as an intersection of constraints." What it shows: Overlapping regions in an output space — schema-valid, grounded, policy-compliant, relevant, within budget — with the intersection shaded and labelled "acceptable." Several sample outputs plotted: three inside the intersection (visibly different from each other), two just outside a single boundary. Caption: "A nondeterministic test does not name a point. It names a region, as the intersection of independently checkable constraints." Placement: End of the composite assertions section.

A deterministic shell around a stochastic core

The techniques so far treat the system as given. It is worth turning the question around: a great deal of the difficulty in testing AI-enabled applications is self-inflicted, caused by asking the stochastic component to do work that ordinary code should do.

If the model is asked to compute a refund amount, then the refund amount is nondeterministic and needs a semantic oracle. If the model is asked to identify the applicable refund rule and conventional code computes the amount from that rule, then the amount is deterministic and needs assert amount == expected. Same product, radically different testability.

The general pattern: let the model handle intent, language, ambiguity, and proposal. Let deterministic code handle everything with a defined right answer.

The shell then does something the model cannot: it rejects. If the model proposes a refund of €740 on an order worth €520, the shell refuses it and the test asserts on the refusal. If the model proposes an action the user is not entitled to take, the shell blocks it and emits an event, and a test asserts that the event fired. The stochastic component becomes a proposer, and the deterministic component becomes both enforcer and oracle.

Two concrete consequences for test design. First, the highest-severity properties — money, permissions, destructive actions — move out of the semantic layer entirely, and can be tested exhaustively with fast, exact tests. Second, the tests that remain in the semantic layer are the ones where semantic judgment is genuinely required, which keeps the expensive part of the suite small.

This is also the answer to a question teams ask when they see how much machinery semantic testing requires: is all of this really necessary? Often, less of it is necessary than expected — after the system has been restructured so that fewer decisions are made in the part that varies.

Mocking: what a mocked test actually proves

Teams reliably err in one of two directions. Either the model is mocked everywhere, and the suite never observes real model behavior at all; or every test calls a live model, and the suite becomes slow, expensive, and unstable enough that people stop running it.

Both errors come from failing to state what a given test is trying to prove.

A mocked-model test freezes the model's output and exercises everything downstream:

model.respond_with({
    "tool": "get_customer",
    "arguments": {"customer_id": "C-1842"}
})

result = orchestrator.handle("Look up my account")

assert crm.calls == [("get_customer", {"customer_id": "C-1842"})]
assert result.state == "awaiting_customer_data"

This is a good test. It runs in milliseconds, it is perfectly reproducible, and it verifies real logic: argument marshalling, tool dispatch, state transitions, error propagation. Write hundreds of them. They are where you test the paths that are hard to trigger with a live model — the tool that times out, the tool that returns an empty result, the malformed function call, the model that returns three tool calls when the orchestrator expects one, the retry that must not duplicate a side effect.

But be exact about the claim it supports:

PROVES:         If the model emits X, the application handles X correctly.
DOES NOT PROVE: The model will emit X.

That second line is where suites go wrong. A team mocks the model returning get_customer with the right ID, watches the test pass, and believes the routing works. Routing is the part that was mocked. The test verified everything except the decision.

The corollary is that mocked tests and real-model tests are not substitutes at different price points. They answer different questions, and a suite needs both. Mocks verify the deterministic shell. Real-model tests verify the core. Neither covers the other.

A practical detail that pays for itself: derive mock payloads from recorded real interactions rather than hand-writing them. Hand-written mocks encode what an engineer assumed the model emits, which drifts from what it actually emits — particularly across model upgrades, where the shape of tool-call payloads can change subtly. A recorded fixture that is periodically refreshed against the live model keeps the mocked layer honest.

Record and replay

Recording executions and replaying them is the single highest-leverage piece of infrastructure for this kind of testing, and it is usually built late and regretted.

What to record, per execution:

trace_id:              tr_9f21c4
timestamp:             2026-07-14T09:12:44Z
model:                 provider/model-name@version
parameters:            {temperature, top_p, max_tokens, seed?}
system_prompt_hash:    sha256:...
prompt_template_id:    support.answer.v7
input:                 <user message, conversation history>
retrieval:
  query:               <expanded query>
  index_version:       kb-2026-07-01
  documents:           [{id, score, chunk_hash}]
tool_calls:            [{name, arguments, result, latency_ms, error}]
model_outputs:         [{raw, finish_reason, usage}]
final_response:        <text>
app_state_before/after:<relevant slice>
config_snapshot:       {feature flags, policy version}

Three fields do disproportionate work. prompt_template_id and system_prompt_hash let you tell whether a behavior change came from your prompt or from the model. index_version and document hashes let you tell whether it came from your knowledge base. Without those, every production investigation begins with an argument about what changed.

Three replay modes, for three different jobs

Exact replay. Every non-deterministic response — model outputs, tool results — is served from the recording. Nothing is called. The application executes against a frozen world.

Use for: debugging application logic, reproducing a customer-reported failure, developing and testing evaluators, verifying that a bug fix changes behavior on the exact inputs that triggered it. This mode is fully deterministic, which means it can run on every commit.

Cannot tell you: anything about current model behavior. The model's answer is a fixture.

Dependency replay. Tool responses, retrieved documents, clock, and external data come from the recording; the model is called live.

Use for: the central question of "has model behavior changed on inputs we care about?" This is the right mode for prompt-change evaluation and model-upgrade evaluation, because everything except the thing under test is held constant. Variance in the result is attributable to the model rather than to a knowledge base that was reindexed on Tuesday.

Costs: real tokens, real latency, and results that vary run to run.

Full re-execution. Everything live.

Use for: end-to-end confidence on a small, carefully chosen set. Measures the system as it actually is, including integration behavior that mocks paper over.

Costs: the most, reproduces the least. Keep the set small and the schedule infrequent.

Most teams need all three, and the mistake is picking one. The suite should be able to say, for any given test, which mode it runs in — and that should be a field in the test definition rather than an accident of how the fixture was written.

Recorded data is production data

A trace archive contains customer messages, account data, retrieved documents, and tool payloads. That is a personal-data store, and it needs the treatment: redaction or tokenization at capture time rather than later, retention limits, access controls, and a documented purpose. Synthetic substitution — replacing real identifiers with stable fake ones while preserving structure and referential integrity — is usually preferable to redaction, because a trace with holes in it does not replay.

One practical note: redact at capture, not at export. A pipeline that stores raw traces and sanitizes on read has a raw trace store, whatever the intention was.

Figure 6 — "Three replay modes." What it shows: The same execution pipeline drawn three times, with each stage shaded to indicate replayed (from recording) or live. Mode 1: everything replayed. Mode 2: dependencies replayed, model live. Mode 3: everything live. A column on the right gives, for each mode, "reproducible / cost / what it can tell you." Caption: "The replay mode is part of the test definition, not an implementation detail." Placement: After the three-mode descriptions.

Controlling variance rather than pretending to remove it

Three distinct activities get confused with each other:

• Eliminating variance — making the system produce identical output.
• Controlling variance — holding constant everything you are not testing.
• Measuring variance — treating the spread itself as an observable.

Elimination is the least available and the most requested. The seed parameter offered by several providers is explicitly documented as best-effort: repeated requests with the same seed and parameters should return the same result, determinism is not guaranteed, and a fingerprint field is exposed so callers can detect backend changes that affect it (OpenAI cookbook). Microsoft's documentation for the same feature is blunter, noting that some variability is commonly observed even with matching seed and fingerprint, and that longer outputs tend to be less reproducible (Azure OpenAI reproducible output). As discussed earlier, a substantial part of the residual variation is a property of how inference is served rather than of sampling.

So: use deterministic settings where they exist, and do not build a suite whose correctness depends on them holding. A test that fails when the model produces a semantically identical answer with different word order was going to break eventually anyway.

Control is where the actual leverage is, and it is ordinary engineering:

• Pin the model version explicitly. Never test against a floating alias.
• Freeze the retrieval corpus and index version for the test environment.
• Inject the clock. "Today" is a dependency.
• Stub or replay tool responses unless the tool integration is what you are testing.
• Fix conversation history explicitly rather than accumulating it.
• Pin the prompt template by identifier and hash.
• Isolate tenant data so one test's writes cannot alter another's reads.

Measurement is the remainder, and it is the subject of the next section. The goal of control is not to make variance disappear but to ensure that whatever variance survives is the variance you are trying to observe:

Reduce irrelevant variance in the lower layers so that the higher layers are measuring model behavior rather than measuring your test environment.

What to mock and what to keep real

The decision is always downstream of a single question: what is this test trying to prove? Answer that, and the rest follows.

Two rows deserve comment. The safety row mocks tools deliberately: a test that verifies the agent would have issued an unauthorized refund should record the attempt, not perform it. Executing destructive operations to verify that they should not have been executed is a category of test failure with an unusually long recovery time.

The agent row uses "simulated with real contracts" — tool implementations that are fake but validate arguments against the real schema and enforce the real preconditions. This catches the large class of agent failures that consist of calling a real tool with arguments it would have rejected, without requiring a live integration.

When one run is not a verdict

A test passes eight times and fails twice. What have you learned?

Not, in general, that the test is flaky. You have learned that the system's behavior on this input has a failure rate somewhere in the vicinity of 20%, measured with a sample too small to say much more. In a deterministic system that observation would be a bug in the test. Here it may be an accurate measurement of the product.

This is the point where a distributional oracle replaces a binary one:

Binary:         the behavior must be correct on this execution.
Distributional: the behavior must be correct in at least N of M executions.

The τ-bench authors formalized a version of this for agents with the pass^k metric — the probability that an agent succeeds on all k attempts at the same task — precisely because single-attempt success rates hide inconsistency. Their reported results showed a substantial gap between single-attempt and repeated-attempt success for function-calling agents of the time, which is the empirical case for measuring reliability rather than capability (Yao et al., 2024).

Enough statistics to make engineering decisions, and no more:

• Small samples are nearly uninformative about rare events. Five runs cannot distinguish a 2% failure rate from a 0% one. If you care about rare failures, either run many more samples or — better — convert the property into a deterministic check that does not need sampling.
• Repetition count should follow severity, not uniformity. Running every test ten times is a way to spend ten times as much for no additional insight on the tests that never vary. Run a small number of high-consequence, high-variance cases many times, and everything else once.
• The observed rate needs a confidence interval, or at least the sample size, attached. "9/10" and "90/100" are different claims. Report both numerator and denominator; never report a bare percentage.
• Trends beat thresholds. A case that went from 19/20 to 14/20 after a prompt change is a signal even if both numbers are above whatever bar you set.

Some behaviors that suit distributional oracles: tool-routing correctness on genuinely ambiguous requests; refusal rate on borderline inputs; task completion on multi-step workflows; ranking stability; format conformance under unusual inputs.

All numbers below are illustrative. There is no universal pass rate, and any article that gives you one is selling something.

- case: ambiguous_routing_007
  runs: 20
  require: correct_tool_selected in >= 18 of 20
  rationale: two plausible interpretations; occasional misrouting is
             recoverable in the next turn

- case: destructive_action_guard_002
  runs: 20
  require: unauthorized_refund_attempted in 0 of 20
  rationale: not a rate; any occurrence is a defect

Those two entries look similar and are fundamentally different, which brings us to the most important distinction in this section.

Some failures must never be averaged away

Take a thousand evaluations. Nine hundred and ninety-five are excellent. Five fail. The aggregate reads 99.5%, which in most engineering contexts is a good number.

Now ask what the five were.

If they are answers that were correct but unnecessarily terse, the aggregate is meaningful and the number is good. If they are five instances of disclosing another customer's account balance, the aggregate is not merely misleading — it is the wrong instrument. There is no rate of unauthorized data disclosure that constitutes passing.

Two categories of requirement, which need different oracles and different reporting:

Population-level quality. Helpfulness, tone, completeness, conciseness, format adherence, routing preference. These are legitimately statistical. They vary, the variation is tolerable, and the right question is whether the distribution is acceptable and whether it is moving. Aggregate them. Track them over time. Gate on trends rather than on individual runs.

Critical-case correctness. Cross-tenant data exposure. Unauthorized or destructive actions. Fabricated financial, legal, medical, or safety information. Actions taken against the wrong account. Failure to disclose a legally required term. These are not statistical properties. They are invariants, and a single violation in a thousand runs is a defect report, not a data point.

The engineering implication is structural. Critical-case properties should be implemented as hard assertions — ideally deterministic ones, enforced in the deterministic shell so that a violation is impossible rather than improbable — and they should never be aggregated into a quality score. If your evaluation harness computes a single number, make sure that number cannot go up because a critical violation was outweighed by a lot of pleasant prose.

A useful discipline: every requirement in the suite carries an explicit severity field with two values that mean exactly this. blocking requirements fail on any occurrence. distributional requirements are measured over a sample. Forcing that choice at test-writing time surfaces disagreements early, in a design discussion, rather than late, in an incident review.

Flaky test, flaky evaluator, or flaky product?

An intermittent failure has at least four possible causes, and treating them identically is how instability gets hidden:

1. Unstable oracle — the assertion is too strict, or depends on something that was never a requirement.
2. Unstable evaluator — the semantic judge returns different verdicts on the same output.
3. Unstable environment — network, rate limits, test data mutated by another test, clock, index reindexed mid-run.
4. Unstable product — the system genuinely behaves differently, sometimes wrongly.

They are distinguishable, and the procedure is mostly a matter of holding one thing constant at a time:

Intermittent failure observed
  │
  ├─ Re-run the evaluator on the SAME recorded output, 10×
  │     verdicts differ → unstable EVALUATOR
  │
  ├─ Exact-replay the failing trace, 10×
  │     application output differs → nondeterminism inside your own code
  │     output identical, verdict identical → the failure is reproducible;
  │       the question is now whether the oracle is right
  │
  ├─ Dependency-replay (frozen context and tools, live model), 20×
  │     outputs vary, some violate the property → unstable PRODUCT behavior
  │     outputs vary, none violate the property → unstable ORACLE
  │
  ├─ Check environment fingerprints across runs
  │     model version, index version, prompt hash, feature flags differ
  │       → unstable ENVIRONMENT
  │
  └─ Cluster the failures
        all failures share an input feature → not flakiness; a real defect
          with a narrow trigger

That last branch is the one most often missed. Failures that appear random frequently are not: they correlate with input length, with a particular document being retrieved, with a specific customer record shape, with a language, or with a time of day. Before concluding that a failure is stochastic, group the failing runs and look for what they have in common. A 20% failure rate that turns out to be 100% on one input category is a much better defect report than "sometimes fails."

And on retries: re-running a failed test is a legitimate diagnostic step, because the difference between "fails once" and "fails five times in five" is information. It stops being legitimate the moment the retry is automatic and its only effect is to turn a red result green. At that point the mechanism has been repurposed from measuring instability to concealing it, and the suite has quietly agreed to stop reporting a class of defect. If retries are configured, record every attempt and surface the attempt count in the report, so that a test passing on the third try is visibly different from a test passing on the first.

Figure 7 — "Instability triage." What it shows: A decision tree with the four causes as terminal nodes, and the discriminating experiment on each edge (replay evaluator on fixed output; exact replay; dependency replay; compare environment fingerprints; cluster by input feature). Caption: "Four different problems that present identically in CI. The experiments that separate them are cheap." Placement: End of this section.

The judge as an oracle — and the judge as software under test

A model-based evaluator is the right tool for a genuinely narrow set of questions: semantic equivalence, instruction adherence, groundedness against a context, relevance, completeness against a rubric, and pairwise comparison between two candidate outputs. For these, nothing cheaper works well.

It also introduces a second stochastic system into the test, positioned as the arbiter of the first. The known behaviors are documented: the MT-Bench study that popularized the approach also catalogued position bias, verbosity bias, self-enhancement bias, and limited reasoning on certain task types, along with mitigations for some of them (Zheng et al., 2023). Practical implications:

• Position bias: in pairwise comparison, evaluate both orderings and count only agreement. A pair where the verdict flips with position is a tie, not a win.
• Verbosity bias: longer answers score better than they should. Control for length, or use rubrics that score named properties rather than overall goodness.
• Self-preference: avoid using the same model family as generator and judge for anything consequential, or at minimum measure the effect.
• Score compression: absolute 1–10 scores cluster and drift. Binary or three-level verdicts on specific criteria are more stable than a scalar quality score.
• Rubric dependence: the judge's prompt is production code. Version it, review it, and treat a change to it as a change that requires re-baselining.

The framing that keeps this honest: an oracle is infrastructure, and infrastructure gets tested. The point is made directly in the EvalGen work on aligning LLM-assisted evaluation with human preferences — model-generated evaluators inherit the weaknesses of the models they evaluate, so they require validation of their own. That paper also names a phenomenon worth knowing about: criteria drift, where people need criteria to grade outputs but discover their real criteria only by grading outputs, so the specification and the evaluator co-evolve (Shankar et al., UIST 2024).

A test suite for your evaluator

Build a labelled set — a few dozen cases is enough to start — where the correct verdict is not in dispute, and run the evaluator against it whenever the judge prompt, judge model, or rubric changes:

Two properties are worth measuring separately. Agreement with human review on the unambiguous cases — reported as a rate, with the disagreements listed, not as a single correlation coefficient. And self-consistency: run the judge repeatedly on the same output and count verdict flips. A judge that disagrees with itself 15% of the time cannot support a test that fails on a single occurrence; it can only support a distributional measure.

The "verbose and wrong" and "terse and right" rows exist specifically to detect verbosity bias, and the "unsupported but true" row exists to check that the judge is evaluating groundedness rather than its own world knowledge. Both catch real evaluator defects with high frequency.

Worked example: a retrieval-augmented assistant

A customer asks: "If I cancel now, do I still get access, and is there a fee?" The answer lives across three knowledge-base documents and the customer's own plan record. Tested as one black box, this is intractable. Tested as four layers, every layer has a usable oracle.

Layer 1 — Retrieval. No generation involved, so no semantic oracle needed. On a labelled set of questions with known relevant documents:

retrieved = retriever.search(query, user=user, k=5)

assert "kb-cancellation-policy" in ids(retrieved)     # required evidence present
assert all(d.tenant == user.tenant for d in retrieved) # entitlement
assert all(d.index_version == PINNED_INDEX for d in retrieved)
metrics.record("recall@5", recall(retrieved, case.relevant_ids))

Retrieval quality is measured, not asserted, except for the two hard constraints: the document containing the answer must be present, and nothing outside the user's entitlement may be. The first is what makes the rest of the pipeline possible; the second is a security property.

Layer 2 — Generation, given fixed context. Retrieval is replayed, so any variance is the model's.

claims = extract_claims(response, schema=CANCELLATION_CLAIMS)

# grounded facts
assert claims.notice_period_days == context_fact("notice_period_days")
assert claims.fee_amount == context_fact("early_termination_fee")
assert claims.access_until == account.current_period_end

# no fabrication: every duration/amount in the text traces to context
assert_all_numerics_grounded(response, context)

# required disclosure
assert claims.fee_amount is not None, "applicable fee not disclosed"

Layer 3 — Citation integrity. Cheap, deterministic, and frequently the first thing to break:

for citation in response.citations:
    assert citation.doc_id in ids(retrieved)          # cited doc was retrieved
    assert citation.chunk_hash in chunk_hashes(retrieved)
    assert supports(citation.chunk_text, citation.claim)  # entailment check

The third line is the one that matters. A citation pointing at a real document that does not actually support the sentence it is attached to is worse than no citation, because it manufactures confidence.

Layer 4 — Variance. The same case, run ten times with retrieval replayed:

runs = [execute(case) for _ in range(10)]
claim_sets = [extract_claims(r.text) for r in runs]

# the facts must be identical across runs
assert len({(c.notice_period_days, c.fee_amount, c.access_until)
            for c in claim_sets}) == 1

# the wording is expected to differ, and that is not asserted on
metrics.record("response_length_spread", spread(len(r.text) for r in runs))

That test says exactly the right thing: the sentences may vary freely; the facts may not vary at all.

Worked example: an agent with account tools

Task: "Find my order from last month and get me a refund for the damaged item." Available tools: search_customer, get_account, get_orders, calculate_refund, draft_email, submit_refund.

Several routes are legitimate. The agent may search by email then fetch orders, or fetch the account first and read the order list from it. It may compute the refund before or after drafting. Encoding one observed sequence produces a test that fails on improvements.

Here is what must hold regardless of route:

t = result.trace
names = [c.name for c in t.tool_calls]

# 1. Subject correctness — every account-scoped call is about this customer
assert {c.arguments["customer_id"] for c in t.tool_calls
        if "customer_id" in c.arguments} == {case.customer_id}

# 2. Precedence — identity established before account data is read
assert first_index(names, {"search_customer", "authenticate"}) \
       < first_index(names, {"get_account", "get_orders"})

# 3. Prohibition — no irreversible action without an explicit confirmation
if "submit_refund" in names:
    assert t.confirmation_turn_index is not None
    assert index(names, "submit_refund") > t.confirmation_turn_index

# 4. Amount correctness — computed by code, asserted exactly
assert result.refund_amount == expected_refund(case.order, case.damage_class)

# 5. Budget — cost containment
assert names.count("calculate_refund") <= 2
assert len(t.tool_calls) <= 8

# 6. Outcome — the task actually finished
assert t.final_state.draft_email_created is True
assert t.final_state.refund_submitted is False   # confirmation not given

Six assertions, none of which mentions a sequence. Assertion 3 is the one that would justify the whole suite on its own: it is the difference between an agent that proposes a refund and an agent that issues one. Assertion 6 pins the negative outcome, which is easy to forget — tests that only assert what should happen are blind to the extra thing that also happened.

For the ambiguous-routing part of this task — whether the agent searches first or fetches the account first — the right oracle is distributional, not binary: record which route was taken across twenty runs and alert on a shift in the distribution rather than failing on either choice.

Worked example: structured extraction

An invoice arrives as a PDF. The system emits JSON. json.loads() succeeding proves almost nothing, and the gap between "parses" and "correct" is where the real defects live.

doc = json.loads(raw)

# structural
jsonschema.validate(doc, INVOICE_SCHEMA)
assert doc["currency"] in SUPPORTED_CURRENCIES
assert isinstance(doc["invoice_date"], str) and parse_date(doc["invoice_date"])

# internal consistency — the highest-yield checks in extraction
assert sum(li["amount"] for li in doc["line_items"]) == doc["subtotal"]
assert abs(doc["subtotal"] + doc["tax"] - doc["discount"]
           - doc["total"]) <= 0.01
assert parse_date(doc["period_start"]) <= parse_date(doc["period_end"])

# groundedness — every extracted value is present in the source
for value in extracted_values(doc):
    assert appears_in_source(value, source_text, normalizers=[
        currency_norm, date_norm, whitespace_norm]), \
        f"{value} not found in source document"

# completeness
assert set(REQUIRED_FIELDS) <= set(doc.keys())
assert not (set(doc.keys()) - set(ALLOWED_FIELDS))

The arithmetic reconciliation and the source-grounding loop together catch most extraction defects without any semantic machinery at all. Both are relational invariants; both are exact; both survive any change in how the model phrases or orders its output. Field order, optional descriptive text, and formatting may vary freely, and nothing in this test notices.

This example is the clearest demonstration of the general principle: nondeterminism becomes manageable in direct proportion to how much of the expected behavior you can express as structured properties.

Oracle brittleness, and what a model upgrade reveals

A new model version arrives. It is better on every benchmark you care about. Forty tests go red.

Before debugging the model, debug the tests, because there is a specific question to answer:

Did the product regress, or did the test encode an implementation detail that was never a requirement?

This is the recurring diagnostic of nondeterministic test design, and a suite that cannot answer it quickly is a suite that will block good changes.

Symptoms of oracle brittleness — the test failing for reasons unrelated to correctness:

• Failures cluster on wording, length, or formatting rather than on facts.
• Trajectory assertions fail because a shorter, valid route was taken.
• Similarity thresholds fail while extracted claims are identical.
• Snapshot diffs are large but claim sets are unchanged.
• Judge scores shift uniformly across the corpus, including on cases the team agrees got better.
• Tests fail only where the new model is more verbose, or more concise.

Symptoms of actual regression:

• Extracted facts disagree with source facts.
• Forbidden tools appear in traces.
• Grounding checks fail: claims no longer traceable to context.
• Refusal behavior changes on cases where the correct behavior is clear.
• Failures cluster on an input category rather than on a surface feature.

The practical value of building the suite the way this article describes is that this triage becomes mechanical. Invariant failures are almost always real. Wording-sensitive failures almost never are. If a model upgrade breaks tests and you cannot tell which kind you are looking at, that is itself the finding: the suite has been asserting on things nobody ever required.

A related habit: when a test is changed to accommodate a model upgrade, record why in the test. "Relaxed because the new model phrases this differently" and "relaxed because it kept failing" look identical in a diff six months later.

Prompt changes without snapshot churn

A prompt edit produces a diff in every response. A snapshot suite will report several hundred changed outputs, which tells you nothing except that you changed a prompt.

The distinction to hold onto:

snapshot difference   = the text changed
behavioral regression = something that was true stopped being true

A prompt-change evaluation should report the second, in roughly this order:

1. Invariant violations — new failures on properties that previously held. This is the gate. Any new violation of a blocking property stops the change.
2. Known-defect status — cases that were fixed must stay fixed. This is the regression corpus, and it is the most valuable asset in the suite.
3. Target-task success rate — the behaviors the change was meant to improve, measured before and after with the same sampling policy.
4. Trace policy conformance — tool choice and sequencing still inside the allowed envelope.
5. Distributional shifts — verbosity, refusal rate, tool-preference rate, latency, token use. Not gates; signals. A prompt change that quietly doubles output length has a cost even if every assertion passes.

Snapshots remain useful, just not as oracles. Keep them for human inspection during review — a side-by-side of ten representative before/after responses tells a reviewer more about a prompt change than any metric — and for change awareness. Store them; do not assert on them.

The anatomy of a nondeterministic test case

Conventional test cases record an input and an expected output. That form has no place to put the two things that matter most here: what is allowed to change, and how the environment was pinned. A definition that does:

id: policy_cancellation_fee_014
intent: >
  When a customer on a plan with an early-termination fee asks about
  cancelling, the assistant must disclose the fee and the access end date.

input:
  message: "If I cancel now do I still have access, and will it cost me?"
  customer: C-1842            # plan: annual_pro, fee applies
  conversation_history: []

environment:
  model: provider/model-x@2026-05-11        # pinned, never an alias
  parameters: {temperature: 0.2, max_tokens: 600}
  prompt_template: support.answer.v7
  retrieval: replay://tr_9f21c4            # frozen context
  tools: simulated                          # real schemas, no side effects
  clock: 2026-07-14T09:00:00Z

allowed_variation:
  - wording, tone, length, sentence order
  - language (any supported locale)
  - presence of an optional closing offer to help further
  - which of the two fee-explanation documents is cited

required_properties:
  - id: fee_disclosed          severity: blocking    oracle: claim_extraction
  - id: fee_amount_correct     severity: blocking    oracle: deterministic
  - id: access_end_date_correct severity: blocking   oracle: deterministic
  - id: all_numerics_grounded  severity: blocking    oracle: deterministic
  - id: addresses_question     severity: distributional oracle: judge

forbidden_properties:
  - id: states_unsupported_period   severity: blocking
  - id: cross_tenant_data_present   severity: blocking
  - id: destructive_tool_invoked    severity: blocking

trace_constraints:
  - get_account called with customer_id == C-1842
  - no tool outside {search_kb, get_account} invoked
  - tool_calls <= 4

sampling_policy:
  runs: 5
  blocking_properties: must hold in 5/5
  distributional_properties: >= 4/5

evaluator:
  judge_model: provider/model-y@2026-04-02
  rubric: rubrics/addresses_question.v3
  self_consistency_measured: 0.94 on 2026-06-30

failure_evidence:
  capture: [input, retrieved_context, full_trace, all_run_outputs,
            extracted_claims, failed_property, judge_rationale,
            environment_fingerprint]

The field that does not exist in conventional test cases, and that carries the most weight here, is allowed_variation. Writing it forces the team to state what is not a requirement — which is the precise thing that string equality assumed and never articulated. In practice, filling in this field is where most of the useful disagreement happens: two engineers will discover they had different views about whether citing either fee document is acceptable, and that disagreement is a requirements gap, not a testing detail.

severity is the second load-bearing field, because it decides whether a property is a gate or a measurement, and that decision should never be implicit.

Figure 8 — "Anatomy of a nondeterministic test case." What it shows: The YAML structure above rendered as a labelled diagram, with the fields grouped into four bands: fixed (input, environment), free (allowed variation), constrained (required, forbidden, trace), and observed (sampling, evidence). Caption: "The conventional test case has the first band and an expected output. Everything else in this diagram is what replaces the expected output." Placement: Directly after the YAML.

An oracle design worksheet

For a single behavior, before writing any code, answer these. It takes about fifteen minutes per behavior in refinement and removes most of the arguments that otherwise happen in code review.

Observation. What can the test actually see? Response text, structured payload, trace, tool arguments, database state, emitted events, latency. If a requirement is not observable through any of these, the requirement cannot be tested — and the correct fix is usually to make it observable rather than to weaken the test.

Stable requirement. What must be true of every acceptable output? Write it as a sentence with no reference to phrasing.

Allowed variation. What may legitimately differ? Be specific and generous. Under-specifying here is what creates brittle tests.

Forbidden variation. What difference would constitute a defect? These become your forbidden properties, and they are usually the shortest and most valuable list on the page.

Oracle. For each property: deterministic code, schema, relational check against source, set membership, range, metamorphic relation, trace policy, judge, or human. Prefer the earliest item on that list that can do the job.

Repetition. Does one execution settle it? If not, how many, and what rate is acceptable — or is this a property where any occurrence is a failure?

Dependency control. What is mocked, pinned, replayed, or live? Which replay mode?

Failure evidence. What must be captured for someone to debug this at 2 a.m. without re-running anything?

Severity. Blocking or distributional. Decide now.

Production feedback. How will a new real-world failure of this kind become a new case here?

The layers that emerge, and what they cost

Nondeterministic systems do not need a new pyramid so much as they need honesty about which layer answers which question. A distribution that tends to work:

No universal proportions; they follow from architecture and risk. A system that only answers questions needs less trace testing than one that moves money. What is not architecture-dependent is the top row's dominance: calling a live model inside a unit test is almost always a design error, because it makes a fast, deterministic check slow, expensive, and capable of failing for reasons unrelated to the code being tested.

The economics are simple enough to state and easy to ignore. A real-model test with five repetitions and a judge call per run costs roughly eleven model calls. Two hundred such tests on every commit is a budget line and a twenty-minute wait. The levers, in order of effectiveness: push properties down into deterministic layers; use dependency replay so tool and retrieval costs disappear; repeat only where severity justifies it; cache aggressively on unchanged inputs; segment the corpus so a small set runs per commit and the full set runs nightly; and select tests by what the change touched — a prompt change needs the generation corpus, not the parser suite.

Failures have to be debuggable

A failing test that reports this is close to useless:

FAIL: test_cancellation_policy
Quality score: 0.71 (expected > 0.75)

It names no defect, suggests no fix, and cannot be triaged without re-running the scenario by hand. Compare:

FAIL  policy_cancellation_fee_014          (3 of 5 runs)

FAILED PROPERTY  fee_disclosed  [blocking]
  Required: the applicable early-termination fee is stated when one applies
  Account:  C-1842  plan=annual_pro  fee=€49.00
  Observed: no fee amount present in response (runs 2, 4, 5)

RUNS   1 PASS   2 FAIL   3 PASS   4 FAIL   5 FAIL

ENVIRONMENT
  model            provider/model-x@2026-05-11
  prompt_template  support.answer.v7  (sha 4c1e…)
  retrieval        replay://tr_9f21c4   index kb-2026-07-01
  clock            2026-07-14T09:00:00Z

EXTRACTED CLAIMS (run 4)
  access_until      2026-09-30   ✓ matches account
  notice_period     30 days      ✓ grounded in kb-cancellation-policy#3
  fee_amount        null         ✗ expected 49.00

CONTEXT
  kb-cancellation-policy#3   retrieved, rank 1
  kb-fees-annual#1           retrieved, rank 4   ← contains the fee
TRACE
  get_account(customer_id=C-1842) → 42 ms  ✓
  2 tool calls, none forbidden

BASELINE   5/5 pass on 2026-07-01, same case, model@2026-04-02
ARTIFACTS  runs/policy_cancellation_fee_014/{1..5}.json

Everything needed for a diagnosis is present, and the diagnosis is nearly automatic: the fee document is being retrieved at rank 4 and the model is not using it consistently. That is a retrieval-ranking problem, not a generation problem, and nobody had to guess.

The general rule: a failure report should name the violated property, show the evidence for the violation, state what varied across runs, fingerprint the environment, and link the artifacts. Anything less and the team will start ignoring failures, which is the actual failure mode of quality systems.

Granularity

The report above is only possible because the assertions are specific. "Is this answer good?" cannot produce it. Breaking quality into named dimensions — was the question answered, was the correct policy applied, was the amount right, was anything unsupported introduced, was the required disclosure present — improves debugging, improves evaluator reliability (narrow questions get more consistent verdicts than broad ones), and makes regression analysis possible, because you can see which dimension moved.

The opposite failure is real too. A hundred micro-assertions that overlap heavily produce a wall of red on any single defect and take longer to maintain than the feature. The useful test: each property should be able to fail alone. If two properties always fail together, they are one property.

Production evaluation is a different instrument

Offline tests and production evaluation answer different questions, and collapsing them costs you both.

Offline suite:  Does this build satisfy the expectations we have encoded?
Production:     How is the system behaving under usage we did not anticipate?

The offline suite is a closed world. Its coverage is exactly what someone thought of. Production is where you discover the question categories nobody imagined, the phrasings that break routing, the documents that retrieve badly, and the failure modes that only appear at the tail of a real distribution.

Production evaluation has its own toolkit — sampled trace review, automated checks on live traffic, user feedback signals, escalation and handoff rates, drift monitoring on input and output distributions — and it is worth keeping the terms distinct. Observability tells you what happened. Monitoring alerts you when a signal crosses a line. Evaluation judges whether behavior was good. Testing verifies encoded expectations before release. Four different things; four different owners, often.

The connection that matters is one-directional and should be a routine: production failures become offline test cases. Not summaries of them — the actual traces, converted into reproducible cases.

From production failure to stable regression test

unexpected production behavior
  → trace captured (with environment fingerprint)
  → failure classified (grounding? routing? retrieval? policy? formatting?)
  → source of nondeterminism isolated (model / retrieval / tool / state)
  → oracle selected (which property was actually violated?)
  → dependencies frozen (replay context and tools; pin model)
  → regression case created, with allowed_variation stated
  → repeated execution to establish the failure rate
  → case enters the suite with a severity

Two stages carry most of the value. Failure classification determines which layer the test belongs in — a retrieval failure tested at the generation layer will produce a confusing test that fails for the wrong reason. And oracle selection is where the discipline lives: the question is not "what did it say?" but "what property did this violate?"

That distinction determines whether the regression test is worth having. A test that captures the response and asserts on it will pass forever after the fix and detect nothing else. A test that captures the violated property — "stated a refund period absent from the retrieved context" — will catch every future instance of that defect class, on inputs nobody has thought of yet.

Figure 9 — "From incident to regression case." What it shows: A pipeline from a production trace through classification, isolation, oracle selection, and dependency freezing, to a test case entering the suite — with a fork partway through showing two possible outcomes: "snapshot of the bad response" (dead end, catches nothing) versus "property that was violated" (generalizes). Caption: "A regression test should preserve the reason the behavior was wrong, not the wrong behavior itself." Placement: End of this section.

One suite, end to end

An AI support assistant with retrieval and a small set of account tools. Roughly twenty conceptual tests, spread deliberately across oracle types. This is the shape of a real suite, not a checklist.

Test 20 is the one most suites omit and the one that keeps the other nineteen meaningful.

What survives of assert actual == expected

Return to the assertion the article opened with.

assert response == expected_response

The instinct behind it was never wrong. It says: there is something that must be true about this execution, and I am going to state it precisely and check it automatically. That instinct is the whole of test automation, and nothing about nondeterministic systems weakens it.

What was wrong was the choice of what to assert. expected_response was never a requirement. It was one acceptable output, captured on one afternoon, standing in for a specification nobody had written down. It worked for a long time because outputs were stable enough that the substitution went unnoticed. Generative components did not break the assertion; they exposed what the assertion had been doing.

The replacement is not weaker. It is longer, and it says more:

The output may vary in wording, length, ordering, and language.

The required facts remain true.
The forbidden claims remain absent.
The permissions remain enforced.
The prohibited actions remain untaken.
The task reaches an acceptable end state.
The execution stays inside the allowed envelope.
And the observed variability, across repeated runs, remains within
the limits we decided we could accept — measured, not assumed.

Every line there is checkable. Most are checkable deterministically. Together they specify a region of acceptable behavior rather than a point, and a test that names a region is doing something the old assertion never could: it distinguishes the differences that do not matter from the differences that do.

That is the real work, and it is design work rather than tooling work. No framework will tell you that a paraphrase is acceptable and a changed refund period is not. No evaluator will decide for you that a fee omission is blocking and a terse tone is a metric. Those are product decisions, made by people who understand what the software is for, and then encoded — precisely, automatically, and repeatably — in exactly the way the original assertion was.

Testing a system that never gives the same answer twice is not an exercise in forcing every execution to look identical. It is an exercise in stating, with enough precision to automate, which differences are harmless and which ones mean the system is wrong.

QAtronic works with engineering teams on test automation and quality architecture for AI-enabled products — oracle design, trace-based agent testing, replay infrastructure, and regression strategies for systems whose outputs are allowed to vary.

Sources

• Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 41(5), 507–525. UCL Discovery
• Chen, T. Y., Kuo, F.-C., Liu, H., Poon, P.-L., Towey, D., Tse, T. H., & Zhou, Z. Q. (2018). Metamorphic Testing: A Review of Challenges and Opportunities. ACM Computing Surveys, 51(1), Article 4. ACM DL
• Segura, S., Fraser, G., Sánchez, A. B., & Ruiz-Cortés, A. (2016). A Survey on Metamorphic Testing. IEEE Transactions on Software Engineering, 42(9), 805–824.
• Liu, H., Kuo, F.-C., Towey, D., & Chen, T. Y. (2014). How Effectively Does Metamorphic Testing Alleviate the Oracle Problem? IEEE Transactions on Software Engineering, 40(1), 4–22.
• Claessen, K., & Hughes, J. (2000). QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs. ICFP 2000. — origin of property-based testing.
• Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. ACL 2020. — behavioral test suites for NLP, including invariance and directional-expectation tests.
• Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024/2025). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045; ICLR 2025. arXiv — goal-state outcome evaluation and the pass^k reliability metric.
• Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023. arXiv — position, verbosity, and self-enhancement biases in model-based evaluation.
• Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. UIST 2024. arXiv — evaluator validation and criteria drift.
• He, H., & Thinking Machines Lab (2025). Defeating Nondeterminism in LLM Inference. thinkingmachines.ai; implementation: batch_invariant_ops
• OpenAI. Reproducible outputs with the seed parameter. OpenAI Cookbook
• Microsoft. How to generate reproducible output with Azure OpenAI. Microsoft Learn
• Hypothesis — property-based testing for Python. hypothesis.readthedocs.io