Two diffs that both turn a build green

Consider two changes proposed by an automated repair mechanism after a nightly regression run. Both are small. Both are syntactically valid. Both make a red test go green. They are presented here in the form an engineer would actually encounter them — as a diff attached to a failed run.

The first:

- await page.getByRole('button', { name: 'Save' }).click();
+ await page.getByRole('button', { name: 'Save changes' }).click();

The second:

- await expect(page.getByTestId('order-total')).toHaveText('$199.00');
+ await expect(page.getByTestId('order-total')).toHaveText('$219.00');

Read as pure text manipulation, these are siblings. Each replaces one string literal with another string literal that was observed in the running application. Each was produced by the same procedure: run the test, watch it fail, look at what the application actually does now, write that down, run it again, confirm green. If your only measurement is the ratio of passing tests to total tests, the two repairs are indistinguishable and both are improvements.

They are not siblings. The first change is a statement about how the test finds a control. The second is a statement about what the product is supposed to charge a customer. The first belongs to the test's implementation, which was always going to drift as markup and copy evolve. The second belongs to the test's specification, which is the only reason the test exists at all. A repair system that treats them as the same category of edit has not made your suite more resilient. It has quietly acquired the right to redefine what your product is supposed to do, one string literal at a time.

The uncomfortable part is that the second diff is not obviously wrong. Maybe pricing genuinely changed. Maybe a new tax rule applies to the fixture's shipping address. Maybe a promotional discount expired on schedule and $219.00 is exactly right. In that case, updating the assertion is correct and necessary maintenance, and refusing to update it produces a false alarm that will be triaged by a human, cost an hour, and end in the same edit. The difficulty is not that assertion changes are always wrong. It is that the repair system cannot tell the difference from inside the running application. Both "the price legitimately changed" and "a rounding defect shipped last Thursday" present identically at the point of failure: the DOM says $219.00 and the test says $199.00.

This is the whole problem in one line. A healer observes the current behavior of the system under test. If it is permitted to update expectations to match current behavior, then any regression that survives long enough to reach the healer becomes, by construction, the new expectation. The test suite stops being an independent description of intended behavior and becomes a lagging transcript of actual behavior. It will be green. It will also be useless for the one thing regression suites exist to do.

Most teams introduce automated repair for a defensible reason. End-to-end suites break constantly for reasons that have nothing to do with product correctness: a component library upgrade rewrites the DOM, a build tool changes generated class names, a designer relabels a button, an A/B framework inserts a wrapper element. The research literature has been describing this problem for well over a decade — WATER (Choudhary et al., 2011) framed broken locators as a repair problem and proposed fixing them by finding a similar locator in the new version; Vista (Stocco, Yandrapally, and Mesbah, 2018) added visual analysis; Similo (Nass et al., 2023) scored candidate elements across many weighted attributes rather than one selector. None of that work is frivolous. Test maintenance is a real cost, and reducing it is a legitimate engineering goal.

But almost all of that work targets a specific, bounded sub-problem: given that this test used to interact with a particular element, find that element again in a changed page. The current generation of agentic tooling is not bounded that way. An agent with file-write access to a repository and a browser session can change a locator, add a step, widen a timeout, swap a fixture, rewrite an assertion, or delete a test — and it will report all of those outcomes the same way, as "test now passing." The technique got more capable without the authority model getting any more explicit.

So the question this article is built around is not "does self-healing work." It is narrower and more useful:

When an automated test repairs itself after a failure, how do we know it fixed the test rather than erased evidence of a product defect?

The argument that follows is not that self-healing is dangerous. It is that self-healing is a change-authority problem wearing the costume of a pattern-matching problem. The interesting engineering question is not how accurately a system can find a replacement element. Similarity scoring is largely solved to a useful degree. The interesting question is which parts of a test file an automated system is permitted to modify without a human, and what evidence it must produce before it does.

Two claims will be defended throughout.

The safest self-healing system is not the one that repairs the most failures. It is the one that knows which parts of a test it is authorized to change, and which failures must remain visible.

And, running underneath it:

A test failure is evidence before it is maintenance work.

What follows is a review. Each section puts a proposed repair on the table, states what failed, states what the repair would change, and asks what authority should be required to accept it. The diffs get progressively more dangerous: from selectors, which are mostly implementation, to assertions, business rules, and regenerated tests, which are specification. The hard cases sit in the middle, around navigation, timing, test data, and accessibility semantics.

What "self-healing" actually names

The phrase is used to describe at least eight distinguishable capabilities with very different risk profiles. Collapsing them into one marketing category is the first mistake, because it makes "we use self-healing tests" an unanswerable statement about a team's exposure.

Locator repair. The test can no longer find an element it previously found. The system searches the current page for the most plausible equivalent and rewrites the selector. This is the classical case and the one with the most published research behind it.

Selector strategy substitution. A related but distinct operation: the system does not just find the same element by a new path, it changes the kind of query used — a CSS selector becomes a role-based query, an XPath becomes a test-id lookup. This can improve a test, and it can also silently change what the query means. .btn-primary and getByRole('button', { name: 'Continue' }) may resolve to the same node today and diverge tomorrow.

Wait and timing adjustment. The element exists but was not ready in time. The system extends a timeout, inserts a wait, or replaces an explicit synchronization point with a longer polling window. This is the category most likely to be misclassified as harmless.

Test-data adaptation. A fixture no longer satisfies the preconditions of the scenario — the seeded customer was deleted, the coupon expired, the account tier changed. The system substitutes a different record that lets execution proceed.

Navigation and path repair. The sequence of pages changed. An interstitial appeared, a step was merged, a route moved. The system inserts, removes, or reorders steps until the flow completes.

Expected-content modification. The text, label, or structure the test asserts on has changed, and the system updates the expected value to what it now observes.

Assertion modification. Broader than expected-content: changing a comparison operator, widening a range, replacing an exact match with a looser predicate, or removing a check entirely.

Generated-test repair and regeneration. Rather than patching, the system re-derives the test from observed application behavior — sometimes an entire spec file.

The risk gradient across this list is not arbitrary, and it is worth naming precisely, because it is one of the two or three load-bearing ideas in this article:

Risk rises sharply as automation moves from changing how a test reaches an observation to changing what the test expects to observe.

Everything on the "reaching" side — locators, waits, navigation mechanics, fixture plumbing — describes the route the test takes through the product to arrive at a checkpoint. Changes there are, in principle, invisible to the user of the product. Everything on the "observing" side — expected text, expected values, assertion predicates, snapshot baselines — describes the checkpoint itself. Changes there are, in principle, changes to the definition of correctness.

The complication, and the reason this article is long, is that the boundary leaks. A locator can encode an expectation. A navigation repair can absorb a UX regression. A timeout increase can conceal a four-fold latency regression. The gradient is real, but it is a gradient, not a wall, and the leaks are where defects hide.

Figure 1 — Reaching versus observing. What it shows: a single test rendered as a pipeline — setup → navigate → locate → interact → synchronize → observe → assert — with the first five stages shaded as "route to the observation" and the last two shaded as "definition of correctness." Arrows from a "healer" box point into stages, thin and light on the left, thick and red on the right. Where it belongs: immediately after this section. Caption: The same edit operation carries different meaning depending on which stage of the test it lands in. Repairs to the route are maintenance; repairs to the observation are specification changes.

The current reference point: planner, generator, and healer

The reason this discussion has become urgent rather than theoretical is that automated repair moved from third-party plugins and commercial platforms into a mainstream open-source runner, packaged as an agent rather than an algorithm.

Playwright ships three test agents: a planner, a generator, and a healer. According to the official documentation, the planner explores the application and produces a Markdown test plan; the generator transforms that plan into Playwright test files; and the healer executes the test suite and repairs failing tests. The agents can be used independently, sequentially, or chained in an agentic loop, and the definitions are installed into a project with npx playwright init-agents --loop=<client>, with supported loops including VS Code, Claude Code, Codex, and OpenCode. Playwright describes the definitions themselves as collections of instructions and MCP tools, and advises regenerating them whenever Playwright is updated.

The documented division of labor matters for the argument here, so it is worth stating precisely rather than paraphrasing loosely.

The planner takes a request, a seed test that establishes a usable environment, and optionally a product requirements document. It produces a human-readable Markdown plan under specs/ containing scenarios, steps, and expected results.

The generator takes that Markdown plan and produces executable Playwright tests under tests/. The documentation notes that it verifies selectors and assertions live as it performs the scenarios, and that generated tests may contain initial errors which the healer can subsequently fix.

The healer takes a failing test name. Per the documentation, it replays the failing steps, inspects the current UI to locate equivalent elements or flows, suggests a patch — the examples given are a locator update, a wait adjustment, or a data fix — and re-runs the test until it passes or until guardrails stop the loop. Its documented output is a passing test, or a skipped test if the healer believes the functionality is broken.

Three observations follow, and none of them are criticisms of Playwright.

First, the artifact structure is deliberately auditable. Plans live in specs/ as reviewable Markdown, tests live in tests/, and a seed test bootstraps the environment. That separation is exactly the kind of structure this article will argue for: the intent lives somewhere a human can read and approve, distinct from the executable representation.

Second, the healer's documented repair categories are drawn from the lower-risk end of the spectrum — locators, waits, data. That is a sensible default, and it is worth noticing that the documentation does not advertise assertion rewriting as the point of the tool.

Third, and most importantly for what follows: the healer's alternative outcome is skipping a test. A skip is not a neutral event. It is a decision that a piece of coverage will stop reporting, made by a system whose evidence for "functionality is broken" is that it could not make the test pass. That is precisely the kind of change that must be loud rather than quiet, and a team that pipes agent output straight into a repository without reviewing skips has given away more than it realizes.

Playwright did not invent self-healing and does not claim to. Locator repair has a long research lineage and a long commercial one — Healenium for Selenium, various vendor platforms, and in-house similarity matchers at large organizations. What is new is not the repair capability but the composition: a system that can explore a product, infer what it should do, write the test, run the test, diagnose the failure, and edit the code. Each of those is a different kind of authority. When one loop holds all five, the question of who is checking whom becomes an architectural question rather than a philosophical one. Everything in this article applies equally to Selenium suites with a similarity-based locator layer, Cypress suites with retry-and-rewrite plugins, and homegrown agents wired up over a browser automation MCP server. The runner is incidental. The authority model is not.

Every test file contains two documents

Open any end-to-end test and you are reading two things at once, interleaved without a separator.

The first is a claim about the product. A signed-in customer with a valid cart can complete checkout and receive a confirmed order number. That claim came from somewhere — a requirement, an acceptance criterion, a contract, a regulation, a conversation in which someone decided what the product owes its users. It is stable across redesigns. It does not care whether the confirm button is a <button> or a styled <a>, whether the route is /checkout or /cart/checkout, or whether the confirmation copy says "Order confirmed" or "Thanks — we've got it."

The second is a mechanism for verifying that claim in the current build. Navigate here, find this control, click it, wait for that response, read this element. It is disposable by design. It should change whenever the product's surface changes, and rewriting it is not an admission of failure.

Self-healing is safe exactly to the degree that it operates on the second document and leaves the first alone. That sentence is easy to write and hard to enforce, because the two documents are not separated by any syntactic marker the tooling can rely on. Consider three repairs in ascending order of ambiguity.

- await page.getByTestId('checkout-submit').click();
+ await page.getByRole('button', { name: 'Place order' }).click();

This is very likely implementation-only. The test still clicks the control that submits the order; it just identifies it differently. Arguably the new form is better, because a role-plus-accessible-name query asserts something meaningful about the control. There is a subtle cost — the test is now coupled to a user-visible string, so a copy change will break it where the test-id would not have — but as a repair it does not touch the claim.

- await expect(page.getByText('Order confirmed')).toBeVisible();
+ await expect(page.getByText('Order received')).toBeVisible();

This is ambiguous, and the ambiguity is not resolvable from the page. "Confirmed" and "received" may be a copywriter's preference. They may also be two different states in an order state machine, in which case the old test asserted that the order reached a terminal accepted state and the new one asserts only that the system acknowledged the request. Same shape of diff. Completely different meaning depending on facts that live outside the DOM.

- expect(response.status()).toBe(201);
+ expect(response.status()).toBeLessThan(500);

This is not maintenance under any reading. The original assertion says the request created a resource. The replacement says the server did not have an internal error. A 200, a 202, a 301, a 404, and a 422 all satisfy the new predicate. This repair does not fix a test; it lowers the bar until the current behavior clears it. It is the clearest possible example of a change that keeps a test executing while destroying its reason for existing.

An automated system looking only at the code sees three edits of similar size. An engineer who knows what the test is for sees one routine change, one open question, and one thing that should never be committed without an argument attached.

Figure 2 — Intent and implementation in one file. What it shows: an annotated Playwright spec with two overlay colors. Lines expressing the claim (assertions on order state, confirmation number, total) in one color; lines expressing mechanism (goto, locators, waits, fixture setup) in another. A margin note marks three lines as "ambiguous — expresses both." Where it belongs: alongside the three diffs above. Caption: Test intent and test implementation share a file with no delimiter between them. Repair policy has to reconstruct a boundary the language does not enforce.

A spectrum, not a switch

Because the boundary is reconstructed rather than given, it is more honest to rank repair categories along a continuum than to sort them into "safe" and "unsafe" bins. What follows is a working ordering, from lowest to highest semantic exposure. It is a way of thinking, not a standard, and reasonable teams will move items around based on what their product does.

Structural locator repair. A generated identifier changed (#mui-4821 became #mui-5107), an element moved inside an equivalent container, a class name was regenerated by a build tool. The element's role, accessible name, and position in the user's mental model are unchanged. This is the closest thing to genuinely free repair that exists.

Locator strategy improvement. Replacing a fragile structural query with a semantic one. Usually beneficial, occasionally a widening: page.locator('.cart-item').first() and page.getByRole('listitem').filter({ hasText: 'Wireless mouse' }) are not the same query even when they return the same node.

Interaction implementation. Scrolling an element into view before clicking, waiting for the enabled state, dismissing an animation, handling a newly added focus trap. Mechanical, and usually a real improvement in test correctness — unless the reason the element was not clickable is that the product broke it.

Synchronization strategy. Replacing an arbitrary sleep with a deterministic wait is an improvement. Replacing a deterministic wait with a longer arbitrary sleep is a concealment. Same category, opposite directions.

Navigation and flow. Adding, removing, or reordering steps. The mechanics are simple; the semantics are not, because the sequence of screens a user passes through is product behavior.

Test data. Swapping fixtures changes the scenario, not the mechanism, whenever the data carries eligibility conditions.

Expected content. Text, labels, counts, formatted values. Now firmly on the specification side.

Assertion logic. Operators, tolerances, predicates, and the presence or absence of checks.

Business expectation. Prices, totals, discounts, entitlements, statuses, permissions. Wholly specification. Nothing an application does at runtime is evidence about what it ought to do here.

Two properties of this ordering deserve emphasis. It is not monotone in confidence — a system can be extremely confident about a business-expectation change and completely uncertain about a structural one. And it is not monotone in diff size — the smallest edits in the list (toBe to toBeLessThan, 201 to 200) sit at the dangerous end.

Figure 3 — Repair-risk spectrum. What it shows: a horizontal axis from "structural locator" to "business expectation," with each category placed on it. A second, independent axis above it shows "typical diff size," deliberately uncorrelated, to make the point that small edits are not low-risk edits. Where it belongs: closing this section. Caption: Semantic exposure and diff size are independent. The most consequential repairs are frequently the smallest.

Exhibit A: the locator that looked innocent

The common shorthand is that locator healing is fine and assertion healing is not. The shorthand is wrong often enough to be worth dismantling, because locator repairs are the ones teams enable first and review least.

Here is a repair that a similarity-based matcher would produce with high confidence:

- await page.getByRole('button', { name: 'Delete account' }).click();
+ await page.getByRole('button', { name: 'Deactivate account' }).click();

Consider what the matcher had to work with. Same role. Same position in the DOM. Same containing section, probably the same styling, quite likely the same event handler wiring at the framework level. Under any reasonable weighting of attributes, this is the best candidate on the page by a wide margin, and it is the correct answer to the question the matcher was asked: which element on this page most closely corresponds to the one this test used to click?

But the test was not about clicking an element. The test was about the product's ability to delete an account — which, depending on jurisdiction and product commitments, may be a legal obligation with a defined data-erasure semantics. "Deactivate" may mean the account is suspended and recoverable, retaining all personal data. If the deletion path was removed or broken and replaced by a deactivation path, the test's failure was the finding. Repairing the locator converts the finding into a passing test that now verifies a behavior nobody asked for.

A second example, more mundane and therefore more likely to actually happen:

- await expect(page.getByText('Payment successful')).toBeVisible();
+ await expect(page.getByText('Payment pending')).toBeVisible();

This is nominally a locator change — getByText is a query. It is also, unmistakably, a change to the payment outcome the test accepts. The syntactic category tells you nothing. Any policy that authorizes edits based on the API being called (getBy* allowed, expect restricted) will wave this through, because the edit is inside a getByText call. Policies keyed to syntax are trivially defeated by tests written in the ordinary style where the locator carries the assertion.

So what should a locator repair actually consider before it is accepted? At minimum:

• Role. Did the element keep its ARIA role? A button that became a div is a different fact than a button that moved.
• Accessible name distance. Not string similarity — semantic distance. "Submit order" to "Place order" is close. "Delete" to "Deactivate" is not, despite a shorter edit distance in some metrics.
• Surrounding context. Same section, same heading, same landmark, same form. An element with a similar name in a different region of the page is a different element.
• Action semantics. What does interacting with it do? A click that fires a DELETE /accounts/:id and a click that fires PATCH /accounts/:id {status:"inactive"} are not interchangeable regardless of visual similarity.
• Observable side effect. Does the resulting state match what the test expects downstream? This is the strongest available signal and the most underused.
• Downstream assertions. If the repaired locator makes the following assertion trivially true — or makes it fail differently — the repair changed the test's meaning even though only one line moved.

The last two points suggest a practical rule that costs nothing to adopt: a locator repair is safest when the assertions after it are unchanged and still discriminating. If a locator repair is accompanied by, or requires, any adjustment to what happens next, it is not a locator repair. It is a behavior change that begins with a locator.

The research on similarity-based localization is careful about exactly this. Similo's evaluation measured whether the algorithm found the correct target element across website versions — it failed on 72 of 598 cases against 146 for the baseline — which is a robustness result about element identification, not a claim that any element it finds is a semantically valid substitute for the purposes of a given test. Those are different questions, and productizing the first as if it answered the second is how locator healing acquires an undeserved reputation for safety.

Pass rate is a proxy, and proxies get optimized

Most organizations that adopt automated repair adopt a metric alongside it, usually some version of suite pass rate or its inverse, maintenance hours. Both are reasonable operational indicators and terrible optimization targets, for a reason that has nothing to do with AI: they are proxies for a goal they do not contain.

The goal of a regression suite is not to pass. It is to produce reliable evidence about whether the product still behaves as intended. Passing is what that evidence looks like when the answer is yes. Treating the observable and the objective as the same thing invites a system to produce the observable by any available route.

An aggressive healer improves every metric a delivery organization typically watches. Pipeline stability goes up. Mean time to green goes down. The flaky-test list shrinks. The QA team's maintenance backlog empties. Developers stop complaining that the suite is noise. Each of these is a real benefit, and each is compatible with a suite whose defect-detection capability has been steadily eroding for two quarters.

The failure is not detectable from inside the metrics, which is what makes it dangerous. A suite that has been healed into insensitivity looks exactly like a suite that has been engineered into robustness. Both are green. Both are fast. Both require little maintenance. The difference only becomes visible when a defect escapes to production and someone asks why the checkout suite did not catch it — and the answer, discoverable only by reading the repair history, is that the assertion which would have caught it was widened eleven weeks ago to resolve a flake.

A measure that becomes a target stops measuring what it measured. The gradient here is unusually steep: repairing is fast, cheap, and produces a visible win, while investigating is slow, ambiguous, and often ends with "nothing was wrong." A system rewarded on repair throughput will drift toward repairing, and an organization rewarded on green pipelines will not push back, because from where it sits the pipeline is green.

The correction is not to abandon the metric. It is to pair it with a second one that moves in the opposite direction under abuse — some measure of the suite's continued ability to detect defects — so that a repair strategy which trades detection for stability shows up as a trade rather than as a win. That is the subject of a later section on measuring healing quality, and it is the single most useful organizational change available here.

Two ways to be wrong

Any automated classifier over failures has two error modes, and they are not symmetric in cost.

False healing. The healer concludes that the test is out of date and edits it. The product was actually broken. The result is a false negative: a defect that existed, was detected by the suite, and has now been removed from the record. The evidence is gone, the build is green, and no human ever saw the failure. The cost is bounded only by what the defect does in production.

Missed healing. The product changed legitimately, the test implementation is stale, and the healer declines to adapt. The result is a false positive: a failing test that requires human triage and ends in a routine edit. The cost is an engineer's time and some erosion of trust in the suite.

Every tuning decision trades these against each other. Raising the confidence threshold, narrowing the allowed change categories, or requiring corroborating evidence reduces false healing and increases missed healing. Loosening any of them does the reverse. There is no setting that eliminates both, and there is no universally correct point on the curve, because the two errors have different costs in different contexts.

The relevant variables are worth enumerating, because teams tend to pick a single global setting and then discover it is wrong for half their suite:

• Blast radius. A hidden regression on a marketing page costs a redesign; one in payment capture costs money and possibly a regulatory conversation.
• Test type. A smoke test proving the app boots tolerates aggressive repair. A regression test written after a specific incident should be nearly unrepairable, because its entire value is sensitivity to one thing.
• Regulatory exposure. Where the test is the compliance evidence, an unreviewed automated edit is an audit finding regardless of correctness.
• Reversibility. A team with per-repair commits and a weekly review can afford more automation than one squashing agent output into a nightly bulk commit.
• Detection elsewhere. If production monitoring catches the same regression in minutes, a hidden test failure costs less than if the suite is the only observer.

A useful consequence: repair policy should be attached to tests, not to organizations. A single global "healing: aggressive" flag is almost always wrong, because a real suite contains both throwaway smoke checks and load-bearing contractual assertions.

Exhibit B: the step that appeared out of nowhere

Navigation repair is where the "reaching versus observing" boundary starts leaking in earnest, because the route a user takes through a product is itself product behavior.

The suite encoded this flow:

Cart → Checkout → Payment → Confirmation

The run fails at the payment step: the card-number field is not present on the page the test arrived at. The healer inspects the current UI, finds a page titled "Recommended for you" with a "Continue to payment" link, and proposes:

  await page.getByRole('button', { name: 'Proceed to checkout' }).click();
+ await page.getByRole('link', { name: 'Continue to payment' }).click();
  await page.getByLabel('Card number').fill(CARD.number);

Mechanically, this is a good repair. The healer correctly identified that an intermediate page had been inserted and correctly identified the control that advances past it. The test passes. The flow completes. Nothing was faked.

And yet almost every interesting question is unanswered. Was the upsell page an intentional product change with an approved design, or did a feature flag default flip in a release nobody audited? Is it skippable, or does it block users who do not interact with it? Does it appear for all users or only some, meaning the test is now coupled to an experiment bucket? Does it introduce a new focus order that traps keyboard users? Was there a conversion-rate requirement attached to the checkout funnel that just acquired a new drop-off point? Does the confirmation flow still deliver the same order payload, or did the upsell add a line item?

None of these are visible from the repair. All of them are visible from the failure. The original failing test was, in effect, a change detector for the checkout funnel: the sequence of screens between cart and confirmation is not what it was. That is a useful signal for a product with a funnel, and it was consumed by a one-line insertion.

The general principle: a repair that adds a step to a user journey is asserting that the new journey is acceptable. That assertion has a source, and the source is a product decision, not a DOM inspection. A healer can reasonably detect the inserted step, characterize it (new route, new page title, blocking or skippable, present for all fixtures or some), and propose the patch. It should not be the thing that decides the new funnel is fine.

There is a cheap and useful middle path here that is worth more than it costs. When a navigation repair is proposed, generate the patch but also emit a structured note that the journey changed, and attach it to the test rather than burying it in a log:

test('guest can complete checkout', async ({ page }) => {
  test.info().annotations.push({
    type: 'journey-change',
    description: 'Interstitial route /checkout/recommendations inserted between checkout and payment (detected 2026-08-11, repair #4471)',
  });
  // ...
});

The test stays green, the pipeline stays fast, and the change remains visible in the artifact a human actually reads. The point is not the specific mechanism — annotations, a report attachment, a label on the pull request all work. The point is that the repair should not be the only trace that something about the product changed.

Exhibit C: the timeout that grew

Timing repairs are the most under-examined category on the list, because a timeout is not obviously part of the specification and adjusting one feels like tuning rather than editing.

- await expect(page.getByTestId('order-confirmation')).toBeVisible({ timeout: 5_000 });
+ await expect(page.getByTestId('order-confirmation')).toBeVisible({ timeout: 20_000 });

The test is now stable. It was flaky, and now it is not. The maintenance burden went down. Every operational indicator improved.

The product may also have become four times slower at confirming orders, and this diff is the only artifact that records it.

Notice the structure of the concealment. The old timeout was, implicitly, a performance assertion: confirmation appears within five seconds. Nobody wrote it as one. Nobody reviewed it as one. But it functioned as one, and the healer just relaxed it — not because anyone decided that twenty seconds is acceptable for order confirmation, but because twenty seconds is what makes the test pass. The new value was derived from the failure, which means the tolerance will keep expanding as long as latency keeps growing, and each expansion will look like a small maintenance edit.

A worse variant:

- const response = await page.waitForResponse(r => r.url().includes('/api/orders') && r.status() === 201);
+ await page.waitForTimeout(10_000);

This is not a tolerance adjustment; it is a removal of synchronization. The original code waited for a specific, meaningful event and captured its status. The replacement waits for the clock. It will pass whether the order request succeeded, failed, retried three times, or was never sent at all. It also converts a fast test into a slow one, which is how these edits eventually get noticed — not because someone spotted the lost assertion, but because the suite got slower.

The distinction to hold onto is between correcting a synchronization strategy and increasing tolerance until the failure stops.

Corrections improve the test's model of the application and are usually safe:

• replacing a fixed sleep with a wait on a real condition;
• waiting for a network response or a specific DOM state rather than a wall-clock interval;
• waiting for an element to become enabled rather than merely present;
• awaiting a settled animation or a completed transition rather than polling blindly.

Tolerance increases assert that slower is acceptable and are not safe by default:

• raising a per-assertion timeout;
• raising the global expect timeout in the config, which silently relaxes every assertion in the suite at once;
• adding retries to a test whose failure was deterministic;
• replacing a precise wait with a longer imprecise one.

A workable policy is to permit synchronization corrections automatically while treating tolerance increases as findings that require a named justification. If the confirmation genuinely takes eighteen seconds now and that is acceptable, someone should say so in writing and the assertion should be updated deliberately — and probably accompanied by a real performance check that lives somewhere other than a timeout value.

It is also worth stating plainly that a timeout is a poor performance test. If latency matters, measure it directly rather than encoding it accidentally in synchronization limits:

const t0 = Date.now();
await expect(page.getByTestId('order-confirmation')).toBeVisible({ timeout: 20_000 });
expect(Date.now() - t0, 'order confirmation latency').toBeLessThan(5_000);

This is not elegant, and a real system would use proper instrumentation. But it makes the implicit explicit: the timeout becomes a safety net for the test, and the latency requirement becomes an assertion that a healer must not touch without authority. Making a hidden expectation explicit is often the cheapest way to protect it from automated repair, because policies can only defend expectations they can see.

Exhibit D: the fixture that was quietly replaced

A refund test fails during setup. The seeded customer cust_4417 returns a 404 — the environment was reset, the record aged out, someone cleaned up test accounts. The healer queries the test API, finds an available customer, and substitutes it:

- const customer = await api.getCustomer('cust_4417');
+ const customer = await api.getCustomer('cust_9902');

Execution proceeds. The test passes.

What the test was actually verifying was this: a customer on the Premium tier, with an active subscription older than 90 days, in an EU billing jurisdiction, who purchased within the last 30 days, is eligible for a full refund without support intervention. Every clause in that sentence was encoded in the identity of cust_4417 and nowhere else. If cust_9902 happens to be a Premium EU customer with a recent purchase, the test still verifies the scenario. If cust_9902 is a Basic-tier US customer on a 12-day-old subscription, the test now verifies that some customer can get some refund, and the actual eligibility rule — the entire point — is untested. The suite is green. The rule is unguarded.

This is the sharpest illustration of a general point: test data is part of the test's semantics, not part of its plumbing. A fixture identifier is a compressed encoding of preconditions. Replacing it by identifier similarity or availability decompresses to something else entirely.

The mitigation is to stop encoding preconditions in identifiers and start encoding them as constraints, so that any substitution — automated or human — has something to satisfy:

const customer = await fixtures.customer({
  tier: 'premium',
  billingRegion: 'EU',
  subscriptionAgeDays: { min: 90 },
  lastPurchaseDaysAgo: { max: 30 },
  refundsUsedThisYear: 0,
  paymentMethod: 'card',
});

Now the properties that matter are visible to the reader, to the fixture service, and to any automated repair. A healer that cannot find a customer satisfying these constraints has learned something worth reporting — either the test environment lacks the data, or the eligibility criteria themselves changed — and it can say so instead of silently picking a substitute. A healer that can find one has performed a genuinely safe repair, because the replacement demonstrably satisfies the scenario's requirements.

There is a second-order benefit worth mentioning. Property-based fixtures make it possible to answer a question that identifier-based fixtures cannot: did the repair preserve the scenario? With cust_4417 → cust_9902 there is no way to check. With a constraint set, the check is mechanical. This pattern — replace implicit knowledge with explicit constraints so that automation has something to validate against — recurs throughout this article and is close to being the general solution to the whole problem, to the extent one exists.

One caution. Constraints can themselves become stale. If the eligibility rule changes from 90 days to 60, the constraint set is now wrong, and a healer that "fixes" it by relaxing subscriptionAgeDays has done exactly what an assertion rewrite does. Fixture constraints derived from business rules deserve the same protection as assertions derived from business rules, because they are the same thing wearing different syntax.

Exhibit E: the button that became a div

This is the example most worth carrying out of the article, because it inverts the usual intuition about which failures are real.

The product previously rendered:

<button type="submit" class="btn-primary">Place order</button>

After a refactor to a component library, it renders:

<div class="btn-primary" onclick="submitOrder()" tabindex="-1">Place order</div>

The test fails, because page.getByRole('button', { name: 'Place order' }) finds nothing. A healer inspecting the page sees an element with identical text, identical styling, identical position, and an obvious click handler, and proposes:

- await page.getByRole('button', { name: 'Place order' }).click();
+ await page.locator('.btn-primary').filter({ hasText: 'Place order' }).click();

Confidence: very high. Visual similarity: identical. Functional outcome for a mouse user: identical. The order is placed, the confirmation appears, the test passes.

The failure was the finding. The checkout submit control is no longer a button. It is not in the accessibility tree as an interactive element. It is not reachable by keyboard tab order because tabindex="-1" removes it. It does not respond to Enter or Space. It announces to a screen reader as unlabeled generic content, if it announces at all. For a user relying on assistive technology, checkout is now impossible to complete — which, for a commerce product in most jurisdictions, is a defect with legal weight attached, not a styling regression.

The role-based locator was doing double duty. It was a way to find the element, and it was an implicit assertion that the element was a button with an accessible name of "Place order." That assertion is the reason the test failed correctly. Replacing the semantic query with a CSS query does not repair the test; it removes the check that caught the regression.

This is the strongest available argument against permissive selector-strategy substitution, and it generalizes into a rule that is easy to implement and hard to argue with:

Repair may move a locator from structural to semantic. It should not move a locator from semantic to structural.

A repair that replaces .checkout-submit with getByRole('button', { name: 'Place order' }) strengthens the test. A repair that goes the other direction weakens it, and the weakening is exactly proportional to the accessibility information discarded. Direction of travel along the semantic axis is a computable property of a diff. There is no reason a repair policy cannot enforce it automatically.

The same reasoning argues for making accessibility structure an explicit assertion rather than an accident of locator choice, so that it fails loudly rather than being repairable away:

await expect(page.getByTestId('checkout-actions')).toMatchAriaSnapshot(`
  - button "Place order"
  - link "Return to cart"
`);

Playwright's aria snapshots capture roles, accessible names, and nesting as YAML, which means a structural accessibility regression produces a diff a human can read in a pull request rather than a locator failure that a healer will interpret as drift. There is an obvious tension: snapshots are themselves baselines, and baselines invite automatic updating, which is the subject of the next exhibit. But an aria snapshot at least fails in the right vocabulary — it says "the button became a generic element" rather than "element not found."

Figure 4 — An accessibility regression, healed away. What it shows: two accessibility trees side by side for the same visual UI. Left: button "Place order" as a focusable node. Right: generic with no accessible name, not in tab order. Below, the proposed locator repair with an arrow showing that it succeeds against the right-hand tree. Where it belongs: immediately after this exhibit. Caption: Visual and functional equivalence for a mouse user can coexist with a complete accessibility regression. A CSS-based repair passes; a role-based test fails. The failure was correct.

Exhibit F: the baseline that was overwritten

Visual regression testing makes the underlying problem unusually literal, because updating a baseline is a single command-line flag.

A screenshot comparison fails. The pixels differ. Playwright supports updating snapshots with --update-snapshots, and modern versions distinguish modes — changed updates only snapshots that differ, all updates everything regardless — with a companion --update-source-method controlling whether changes are written as a patch file or directly into source. These are well-designed controls, and they make the operation trivial.

Trivial is the problem. When accepting the current rendering costs one flag and investigating costs twenty minutes, the flag wins under deadline pressure, and it wins permanently: once a baseline is overwritten, the regression is not merely undetected, it is encoded as correct. Every future run will confirm it. Unlike a hidden functional bug, which a later test might catch, a wrongly accepted visual baseline actively defends the defect.

The reasoning should be:

visual difference → evidence → classify → decide

not:

visual difference → update baseline

Automatic baseline updating is defensible in specific situations: approved design-system rollouts, where a reviewed change to shared tokens is expected to alter every screen and the mass update follows a decision rather than a discovery; environment-driven rendering noise, which is better fixed by pinning the environment but at least is not a semantic claim; and deliberate wholesale re-baselining after a framework or browser upgrade, performed as an explicit, dated operation with someone's name on it rather than as a side effect of a nightly run.

Review is essential when the difference is localized rather than global — a single component changed while everything else is stable is exactly the signature of an unintended regression — when it touches a region carrying meaning (price, status, error state, legal copy), when it appears in only some viewports or browsers, or when nobody can name the change that caused it. That last one is the most useful heuristic available: if no one can point to the intentional change that produced the visual difference, the difference is a finding.

A practical arrangement that works: baselines live in version control, updates are never produced by the same run that detected the difference, and the update commit contains the before image, the after image, and a one-line reason. That is not bureaucracy; it is the minimum needed to answer "why does the product look like this?" six months later.

Exhibit G: the assertion

Everything so far has been a case where a repair incidentally changed meaning. Assertion repair is the case where changing meaning is the whole operation.

An assertion is the point in a test where an engineering organization wrote down what it believes the product owes its users. It is the closest thing most codebases have to an executable specification. Everything above it — navigation, locators, fixtures, waits — exists to get the process into a state where that claim can be evaluated.

Compare:

- await page.locator('.submit-btn').click();
+ await page.getByRole('button', { name: 'Submit' }).click();

with:

- await expect(page.getByTestId('price')).toHaveText('$99.00');
+ await expect(page.getByTestId('price')).toHaveText('$109.00');

or:

- expect(order.status).toBe('approved');
+ expect(order.status).toBe('pending');

The first changes how the test reaches its checkpoint. The second and third change what the checkpoint is. A system with unrestricted authority over assertions can, over enough iterations, converge the entire specification onto whatever the application currently does — and it will do so while every dashboard reports improvement, because each individual edit resolved a genuine failure.

The asymmetry is worth making precise. When a locator repair is wrong, the test usually fails again quickly: the wrong element does not produce the expected downstream state, and the failure resurfaces. Bad locator repairs are self-correcting to a useful degree. When an assertion repair is wrong, the test passes forever. There is no downstream check on an assertion; it is the downstream check. Errors in this category do not surface. They accumulate.

That said, "never touch assertions automatically" is too blunt, and pretending otherwise makes the policy easy to dismiss. Legitimate cases exist.

Pure copy changes with no state semantics. A confirmation banner reads "Your order is confirmed" instead of "Order confirmed." If the string appears in a localization catalog that a human already reviewed, and the underlying order status assertion is untouched, updating the expected text is genuine maintenance.

Formatting and locale drift. $1,299.00 becoming $1,299 is a presentation change, provided the numeric value is asserted separately.

Non-deterministic identifiers. An order number that changes every run should never have been an exact-match assertion in the first place; converting it to a pattern is a repair of a badly written test.

Even in these cases, the question of whether the change should be automatic has a different answer than the question of whether it is correct. The distinguishing feature is whether there is an external record of the change. If the string came from a reviewed localization file, an approved design spec, or a merged pull request, the healer is corroborating a decision someone already made. If the string's only provenance is "the page says this now," the healer is making the decision, and the fact that the decision is probably right does not make it the healer's to make.

A design pattern that removes most of this friction: separate the semantic assertion from the presentational one, and let policy apply differently to each.

// Presentational — a copy change here is maintenance.
await expect(page.getByTestId('order-banner')).toHaveText(/order (confirmed|received)/i);

// Semantic — this is the specification. A change here is a product decision.
await expect.poll(() => api.getOrder(orderId).then(o => o.status)).toBe('confirmed');
await expect(page.getByTestId('order-total')).toHaveText(formatCurrency(expectedTotal));

The UI-text assertion is now explicitly a check on copy, and can be repaired liberally. The state assertion reads from an independent source — the order API, not the page that renders it — and cannot be satisfied by a rendering change. Splitting these two apart is one of the highest-leverage things a team can do before enabling automated repair, because it converts an ambiguous edit into an unambiguous one.

Figure 5 — Where the specification lives. What it shows: a test decomposed into layers, with an arrow labeled "healer's authority ceiling" drawn between the synchronization layer and the assertion layer. Above the line, a second arrow labeled "requires external corroboration" points from assertions out to sources of truth (requirements, API contract, design spec, feature flag config). Where it belongs: closing this section. Caption: Repairs below the line are derivable from the application. Repairs above it require evidence the application cannot provide about itself.

Where does an expected result come from?

The previous section ended on a claim that needs defending: an application cannot supply evidence about what it ought to do. This is the crux of the entire argument, so it is worth being precise about it.

Legitimate expected results have external provenance. A partial list of sources, roughly in descending order of authority:

• Explicit product requirements and approved acceptance criteria. Someone with the authority to define the product wrote down what it should do.
• API contracts and schemas. An OpenAPI document, a protobuf definition, a GraphQL schema. These are machine-readable and versioned, which makes them unusually good corroborating evidence.
• Business rules held in a rules engine or configuration. Discount tables, eligibility matrices, tax rules.
• Design specifications. Component definitions, interaction specs, content guidelines.
• Feature-flag configuration. Often the actual answer to "why did behavior change," and frequently the fastest way to distinguish an intentional change from an accident.
• Regulatory and contractual obligations. Retention periods, disclosure requirements, accessibility conformance targets.
• A production baseline that was previously validated, which is the weakest of these but still external to the build under test.

Notice what is absent from the list: the behavior of the build being tested. That exclusion is not pedantry. It is the definition of a test.

Consider the concrete case:

Expected loyalty discount: 20%
Observed loyalty discount: 10%

A repair system with unrestricted authority reasons as follows: the assertion expects 20%, the application produces 10%, the assertion is out of date, update it. The reasoning is locally coherent and globally catastrophic. It is indistinguishable from the reasoning it would apply if a developer had inverted a conditional, if a config deploy had failed halfway, if a currency conversion had been applied twice, or if the loyalty tier lookup was returning the wrong tier for EU customers.

The only thing that distinguishes "the discount policy changed" from "the discount calculation broke" is a fact that lives outside the running application. That fact might be a ticket, a merged change to a pricing configuration file, a marketing decision recorded in a document, or a product owner's confirmation. Absent one of those, the correct output of the repair system is not a patch. It is a report:

FINDING  loyalty-discount.spec.ts › premium tier receives 20% discount
         Expected 20%, observed 10%.
         No corresponding change found in pricing-config (last modified 2026-06-02),
         requirements index, or open change requests referencing loyalty discount.
         Repair withheld: business-expectation change without external corroboration.

The general rule, and probably the single most important sentence in this article:

A healer must never derive a new expected result solely from current application behavior. Doing so converts every surviving regression into the new specification.

The corollary is uncomfortable but true: for the class of expectations that matter most — prices, entitlements, permissions, statuses, calculations — a healer without access to external sources of truth has nothing useful to contribute beyond detection. That is not a failure of the healer. Detection is valuable. It is simply not repair.

Corroborated repair, and why it is harder than it looks

If external evidence is what separates maintenance from specification change, the obvious next move is to give the repair system access to that evidence. This is a promising direction worth describing carefully, including its limits.

The shape is straightforward:

test fails
  ↓
collect evidence (DOM, aria tree, network, trace, logs)
  ↓
generate candidate repair
  ↓
classify repair category and semantic impact
  ↓
if category requires corroboration:
      search authoritative sources for a matching change
      ↓
      corroborated? → propose with evidence attached
      not corroborated? → escalate as finding
  ↓
validate, then apply or escalate

The corroboration step is where the value is. Concretely, a system might check whether a proposed change to an expected order status corresponds to a change in the order-service OpenAPI schema; whether a changed button label appears in a localization file modified in a merged pull request; whether an inserted navigation step corresponds to a feature flag that was enabled in the target environment; whether a changed discount matches a diff in the pricing configuration repository; whether an accessibility structure change was part of an approved component library upgrade.

When corroboration succeeds, the repair arrives with an argument attached: the label changed because commit a3f81c updated en-US.json, reviewed by two people, merged Tuesday. That is a materially different artifact from a patch that says "the page says this now." It is reviewable in seconds rather than minutes, which is what makes higher-risk categories affordable to automate at all.

The limits are real and should not be minimized.

Sources of truth are frequently stale. A repair rejected because the requirements say 20% may be rejected against a document nine months old, during which the policy changed twice.

They are incomplete and ambiguous. Error message wording, list ordering, double-submit behavior — none of it is specified anywhere, and all of it is testable. "The user should be notified of the order status" does not tell you whether "Processing" and "Confirmed" are interchangeable.

They can be gamed. If corroboration unlocks automatic repair, editing the requirements document becomes the fastest path to a green pipeline.

Matching is itself an inference. Deciding that a schema change "corresponds to" a proposed assertion change is a judgment, usually made by a model, and it can be wrong in both directions.

None of this makes corroboration useless. It makes it a strong signal rather than a proof. The practical framing: corroboration should be able to move a repair from "escalate" to "propose with high priority for a fast human review," and it should be able to move a repair from "propose" to "reject and investigate" when the sources actively contradict the change. It should rarely be sufficient on its own to move a business-expectation change all the way to automatic application. The asymmetry is deliberate — corroboration is more trustworthy as a reason to stop than as a reason to proceed, because contradicting evidence is dispositive in a way that consistent evidence is not.

Categories where the policy should be strict regardless

Some test categories deserve much tighter repair permissions than their diff size suggests, because the failure mode is not "a defect escapes" but "a control that was supposed to exist stopped existing."

Consider an authorization test that verifies a standard user cannot reach the admin console. It fails because the assertion await expect(page.getByRole('link', { name: 'Admin' })).toHaveCount(0) now finds one. A healer trying to make this test pass has a menu of terrible options: update the expected count, change the locator to something that finds nothing, or skip the test on the grounds that the functionality appears broken. All three are catastrophic, and the third is the documented behavior of a healer that concludes functionality is broken.

Or: a test verifying that a destructive action requires confirmation fails because the dialog no longer appears. The mechanical repair — remove the now-obsolete dialog-handling step — produces a passing test for a product that deletes data on a single click. Or: a test asserting that a protected endpoint returns 403 for an unauthenticated request fails because it now returns 200, and widening the assertion to toBeLessThan(500) makes the security control undetectable.

The pattern across all three is that the absence of a control is exactly what these tests are looking for, and repair mechanisms are built to route around absence. A healer's core competence — finding an alternative path when the expected one is missing — is precisely the wrong instinct here.

The categories that warrant strict policy: authentication, authorization and role boundaries, payment and financial calculation, destructive or irreversible actions, privacy controls and consent, data retention and deletion, audit logging, and anything carrying explicit regulatory obligation. For these, a defensible default is that no automated repair applies without human approval, including locator repairs, because in these tests the locator frequently is the assertion. The volume is low enough that the review cost is negligible, and the downside of getting it wrong is not comparable to anything else in the suite.

Confidence is a measurement; authority is a permission

Repair systems produce scores. Similarity scores, model confidence, match probabilities. These numbers are useful and they are routinely asked to carry weight they cannot support, because they answer a different question than the one that matters.

Confidence answers: how sure am I that this is the right replacement?

Authority answers: am I allowed to change this part of the test?

A system can be 99% confident that the control formerly labeled "Cancel subscription" is now labeled "Pause membership." The confidence is well founded — same position, same styling, same container, same handler. It is also irrelevant to whether the change should be applied, because canceling and pausing are different product operations with different billing consequences, and the high confidence is evidence that the element corresponds, not that the behavior does. Confidence measures correspondence. It says nothing about equivalence.

Treating a threshold as an authorization gate produces exactly the wrong behavior at exactly the wrong moment: the repairs a system is most confident about are frequently the semantically loaded ones, because semantic changes tend to preserve structure. A button that changed from "Delete" to "Deactivate" is structurally identical and semantically different, which is the worst possible combination for a confidence-gated system.

A decision should therefore be a function of several inputs, of which confidence is one:

• confidence in the candidate;
• repair category on the spectrum described earlier;
• semantic distance between old and new, which is not string distance;
• test criticality, ideally declared on the test itself;
• side-effect risk of the action being repaired — does it move money, delete data, change permissions;
• corroborating evidence from sources outside the application;
• policy, which is the only input that can say "no" regardless of the others.

The practical form of "policy" is a declared change budget: an explicit statement of which categories automation may modify, under what conditions. Here is an illustrative, vendor-neutral policy — not a product configuration, and not a recommendation to copy verbatim:

# ILLUSTRATIVE / VENDOR-NEUTRAL repair policy. Not a product schema.
version: 1

defaults:
  locator.structural:        auto            # generated ids, class churn, container moves
  locator.semantic_upgrade:  auto            # structural -> role/name query
  locator.semantic_downgrade: prohibited     # role/name query -> css/xpath. Never.
  locator.accessible_name:   review          # name changed => possible behavior change
  interaction.mechanics:     auto            # scroll into view, wait for enabled
  sync.strategy_fix:         auto            # sleep -> deterministic wait
  sync.tolerance_increase:   review          # requires named justification
  navigation.step_added:     review
  navigation.step_removed:   prohibited      # a vanished step is a finding
  test_data.constrained:     auto            # replacement satisfies declared constraints
  test_data.identifier:      prohibited      # opaque id swap: constraints unknowable
  expected_text.presentational: review
  expected_value:            prohibited
  assertion.predicate:       prohibited
  assertion.removal:         prohibited
  visual_baseline:           prohibited
  test.skip_or_quarantine:   review          # must create a tracked issue

overrides:
  - match: { tag: ["auth", "payments", "privacy", "destructive"] }
    all_categories: review                   # including locators
  - match: { tag: ["smoke"] }
    navigation.step_added: auto
    expected_text.presentational: auto
  - match: { tag: ["incident-regression"] }
    all_categories: prohibited               # this test exists to catch one thing

escalation:
  on_prohibited: open_finding                # not "fail silently", not "skip"
  on_review: open_pull_request_with_evidence
  competing_candidates: open_finding         # ambiguity is information

Three things about this shape are worth noticing. Policy is declared per category rather than globally, so a team is not forced to choose one aggressiveness setting for a heterogeneous suite. Criticality is expressed as tags on tests, which means the people who write the test decide how much protection it gets. And the escalation section is as important as the permissions: the answer to "not allowed" is produce a finding, never quietly do nothing and never skip the test.

Explicit permissions beat unrestricted write access for a reason that has nothing to do with distrusting the model. A general-purpose agent with repository write access has no way to know that pricing.spec.ts encodes a contractual obligation while nav-links.spec.ts encodes nothing at all. That knowledge exists in the organization, not in the code, and a policy file is the cheapest available mechanism for transferring it.

Repair, regeneration, and the circular oracle

There is a category difference between patching a line and re-deriving a test, and it is easy to miss because both are described as "fixing the test."

Patching starts from an existing artifact that encodes a past decision and makes a minimal change to it. The original intent survives by default; the burden is on the change to justify itself.

Regeneration starts from the current application and produces a new artifact that describes what the application does. The original intent survives only if it happens to coincide with current behavior. The burden has inverted.

The consequence, stated as a sequence:

requirement written
  ↓
test encodes requirement
  ↓
product changes incorrectly
  ↓
test fails  ← this is the system working
  ↓
agent observes current product behavior
  ↓
new test generated from observation
  ↓
suite green
  ↓
regression is now the specification

Nothing in that chain is a bug in the agent. Each step is the agent doing its job. The defect is in the composition — specifically, in allowing the source of expectations to be the artifact under evaluation.

This is a circular oracle. A test oracle is whatever decides whether observed behavior is acceptable; the difficulty of obtaining good oracles is one of the oldest problems in software testing. When an agent derives the oracle from the system under test, the resulting check is not a verification. It is a consistency check between the application and a recording of itself, and it can only fail if the application changes again.

Which is not to say it is worthless. Behavior-derived tests are valuable for:

• initial scaffolding, where the alternative is no coverage and a human reviews before anyone relies on it;
• exploratory mapping of an unfamiliar or undocumented system;
• smoke and availability checks, where the requirement really is "what worked yesterday still works";
• change detection — a behavior-derived test is a tripwire, and tripwires are valuable as long as nobody mistakes them for specifications;
• characterization tests around legacy code slated for refactoring, where freezing current behavior is the explicit goal.

It is insufficient for regression guarantees on business-critical paths, for validating rules that exist independently of their implementation, for compliance evidence, and for anything where "what it does now" and "what it should do" might have diverged before the test was generated. The failure mode is not that generated tests are bad. It is that a generated test's authority is bounded by the correctness of the build it was generated from, and that bound is invisible in the test file.

This is where the three-agent decomposition turns out to be doing real work. Playwright's planner produces a Markdown plan under specs/ — human-readable, reviewable, versioned — and the generator turns that reviewed plan into tests. Intent is captured in one artifact and mechanism in another, with a review point between them. That structure is exactly what prevents the circularity, provided the review actually happens. If the plan is generated, accepted unread, converted to tests, and subsequently healed against the application, the whole loop closes and the human is nominally in it while contributing nothing.

The governance question that follows applies to every agentic testing system, not to any particular one:

Should a single automated system be permitted to observe the product, infer expected behavior, write the test, modify the test when it fails, and judge that the modification is correct?

Five distinct authorities. The argument for separating them is not that models are unreliable; it is the same argument that separates the developer from the code reviewer and the trader from the settlement desk. Independence is a property of the arrangement, not of the participants' competence. A reasonable separation looks like: planning produces a reviewed artifact with a human approval gate; generation is bounded by that artifact and may not invent expectations absent from it; healing may modify implementation freely within policy but may not modify anything traceable to the plan without escalating; and judgment of whether a repaired test still verifies the plan is made against the plan, not against the application.

Figure 6 — The circular oracle. What it shows: two loops. The upper loop — requirement → plan → test → verdict — with the requirement outside the system boundary. The lower loop — application → observation → generated expectation → verdict — entirely inside the system boundary, drawn as a closed circle with no external input. Where it belongs: here. Caption: When expectations are derived from the system under test, the check reduces to a consistency comparison between the application and a recording of itself.

The repair record, and why the original failure must survive it

When automation modifies a test, the modification should be an artifact, not an event that happened inside infrastructure. The distinction is practical: an artifact can be read, questioned, and reversed six weeks later by someone who was not there.

The failure mode to design against is specific. A test fails at 02:14. A healer replays it, finds a candidate, patches, reruns, passes, commits. By 09:00 the suite is green and the only record is a commit titled "fix flaky test." The DOM at the moment of failure is gone. The network activity is gone. The screenshot is gone. When the same test fails again in three weeks — or when a customer reports a bug that this test should have caught — the question why did the original test fail? is no longer answerable. The repair destroyed its own justification.

So: preserve first, repair second. Before applying anything, the system should capture the original test source at the failing commit, the exact failing step and locator, the error, a screenshot, a DOM snapshot, the accessibility tree, network requests and responses around the failure, console output, the execution trace where available, the environment and application build identifiers, and the candidate repair with its supporting evidence. In Playwright this is largely a configuration decision rather than an engineering project — trace: 'on-first-retry' and the associated screenshot and video settings produce most of it — and the important part is that these artifacts are retained beyond the run that produced them and linked from the repair record, not garbage-collected when the pipeline goes green.

Traces deserve a specific mention because they answer the classification question better than any other single artifact. A trace shows the sequence of actions, the DOM state before and after each one, network activity, and console output on a common timeline, which makes it possible to distinguish failure classes that look identical from a stack trace: an element that never existed versus one that existed and was replaced; a click that landed on the wrong node versus one that landed correctly and produced no effect; a request that was never issued versus one that returned a 500; a timing failure where the element appeared 200 ms after the deadline versus one where it never appeared at all. Those distinctions are precisely the input a classifier needs, and they are the reason evidence collection should precede repair rather than being a debugging aid consulted after the fact.

The repair record itself should be readable without opening a tool. A workable shape:

REPAIR RECORD  r-2026-08-11-4471
────────────────────────────────────────────────────────────────
TEST            tests/checkout/place-order.spec.ts › guest checkout completes
TAGS            checkout, revenue-critical
BUILD           app 2026.8.11-rc3   suite @ 9f2c1ab   env staging-eu

FAILURE
  step          click "Place order"
  error         locator resolved to 0 elements
  class         element_not_found (candidate: locator_drift | ui_regression)

EVIDENCE
  aria-tree     button "Place order" absent; generic "Place order" present, tabindex=-1
  dom-diff      <button class="btn-primary"> → <div class="btn-primary" onclick=...>
  network       no request issued (expected POST /api/orders)
  trace         artifacts/r-4471/trace.zip
  screenshot    artifacts/r-4471/failure.png

CANDIDATE REPAIR
  diff          - getByRole('button', { name: 'Place order' })
                + locator('.btn-primary').filter({ hasText: 'Place order' })
  category      locator.semantic_downgrade
  confidence    0.97   (visual + position + text identical)

IMPACT ANALYSIS
  role change            button → generic          YES
  accessible name lost   YES
  keyboard reachable     NO (tabindex=-1)
  downstream assertions  unchanged
  detection strength     accessibility regression no longer detectable

CORROBORATION
  design-system PR #2210 "migrate to <Pressable>"  — found, does not
  document removal of button semantics or tabindex

DECISION
  policy        locator.semantic_downgrade: prohibited
  outcome       REPAIR WITHHELD — finding raised (QA-8842), test remains failing
  rationale     high confidence in element correspondence; loss of accessible
                semantics is the failure, not an obstacle to it

Three properties make this record useful rather than ceremonial. The confidence is high and the decision is rejection, which demonstrates that the two are independent. The impact analysis names what the repair would cost in detection terms rather than only what it would fix. And the outcome is a tracked finding with the test left red — the system did something, and what it did was refuse.

Records like this also aggregate into the only reliable way to detect that a repair strategy has gone wrong. A single bad repair is invisible; a cluster of expected_text repairs in one service, or of tolerance increases in one product area, is a signal about the product rather than the tests. Retaining test identifier, failure class, repair category, confidence, corroboration result, approver, resulting commit, and subsequent pass history makes that analysis mechanical. The question worth asking of that dataset quarterly is not how many repairs were applied, but whether any escaped defect touched code paths where a repair had recently reduced detection.

Classify before repairing

A great deal of harm follows from a single ordering mistake: generating a candidate repair before deciding what kind of failure occurred. Once a plausible patch exists, the framing has already shifted from what happened to does this fix it, and the second question is much easier to answer.

The order that keeps the evidence in front:

failure
  ↓ collect evidence (before any modification)
classify likely cause
  ↓ implementation drift | environment | intentional change | regression
generate candidate repair(s)
  ↓
assess category and semantic impact
  ↓
check policy and corroboration
  ↓
apply | propose | escalate as finding
  ↓
validate the outcome

Classification is also where a persistent confusion should be cleared up: flakiness and staleness are different problems, and repair is the wrong tool for one of them. A stale test fails deterministically because the product changed. A flaky test fails intermittently for reasons rooted in nondeterminism — race conditions between the test and the application, shared state across tests, ordering dependencies, network variance, time and timezone sensitivity, or nondeterministic data. The literature on flaky tests is consistent that these have distinct root causes requiring distinct fixes.

The signature is easy to check and rarely checked: a stale test fails on every run at the same step; a flaky test fails on some runs. Repairing a locator on a flaky test may make it pass, but the passing is a coincidence of timing, and the underlying race is still there — now with a modified locator that nobody can explain. Worse, the standard mitigations for flakiness (retries, longer timeouts, quarantine) are all available to a healer and all of them convert an intermittent signal into no signal. A retry that hides a real race condition in order submission is not a stabilized test; it is a production incident with a delay fuse.

The practical policy: before any repair, check the failure's history. Deterministic failure at a fixed step is a candidate for repair. Intermittent failure is a candidate for root-cause investigation, and the correct automated action is to collect evidence across runs and report, not to patch.

Decision-making across categories can be summarized, with the caveat that every cell is a default a team should argue with rather than a rule:

Figure 7 — Confidence against authority. What it shows: a scatter plot with confidence on one axis and semantic impact on the other, with repair categories placed as points. A shaded region in the high-confidence / high-impact quadrant labeled "the dangerous quadrant — where a threshold-gated system does the most damage." Where it belongs: beside the matrix. Caption: Confidence and authority are orthogonal. The repairs a system is surest about are often the ones it should be least free to apply.

A repaired test that passes has proved almost nothing

The weakest link in most repair workflows is the acceptance criterion. The healer patches, reruns, sees green, and treats green as validation. But green was the objective, which makes it circular as evidence: the patch was selected because it produces green. A test that passes after being modified to pass has demonstrated only that the modification worked.

Several techniques give a repair something closer to real validation, in rough order of cost.

Differential execution. Run the original and repaired tests against the same build and compare the full evidence trail rather than the verdicts: which requests were issued and with what payloads, what application state resulted, which assertions ran against what values, and how the DOM and accessibility tree differed at each checkpoint. The informative outcome is not "one passed and one failed" — that was already known — but where the two executions diverged. Divergence only at the changed locator, with convergence immediately after, is a concrete argument that the repair is implementation-only. Divergence in the requests issued or the resulting state means the repair changed behavior, whatever the diff looks like. This does not resolve semantic questions — it cannot tell you whether "Deactivate" substitutes for "Delete" — but it reliably answers whether the observable path through the product stayed the same, which is more than a rerun does.

Sensitivity comparison. A repair can preserve execution while degrading the test's ability to detect anything. The clearest case:

// before
await expect(page.getByTestId('order-total')).toHaveText(formatCurrency(expectedTotal));

// after
await expect(page.getByTestId('order-total')).toBeVisible();

Perfectly stable. Nearly worthless. The check on the number was the reason the assertion existed; what remains verifies that a DOM node is on screen. A static comparison of pre- and post-repair test source can flag most of this class mechanically, and doing so is cheap: count assertions before and after, detect predicate widening (toBe → toContain, exact value → truthiness, equality → inequality), detect increases in timeouts and retries, detect removal of steps, detect movement from semantic to structural queries, detect newly introduced expect.soft where a hard assertion used to be. Any repair that reduces the number or specificity of checks should be treated as a finding regardless of how confidently it was produced. The rule generalizes: a repair may change how a test observes; it should not reduce how much a test observes.

Mutation reasoning. The most direct answer to "does this test still work" is to break the product deliberately and see whether the test notices. Mutation testing is an established idea with a long research history, and it maps onto repair validation almost perfectly: the question would the repaired test still catch the defect it was written to catch? is a mutation question. In an end-to-end context the mutants are behavioral rather than syntactic — remove the authorization check on an endpoint, alter a price by 10%, skip the confirmation dialog, return the wrong order status, drop a required field from a submission, return a stale cached response. If the repaired test still passes against a build carrying one of those mutations, its detection power has measurably decreased, and the number of surviving mutants is a usable before-and-after measure.

Full mutation analysis over a browser suite is expensive and often impractical at scale — each mutant requires a build and a run. The affordable version is targeted: maintain a small set of mutations per critical flow, run them on a schedule rather than per commit, and run the relevant ones specifically when a repair touches a critical test. Even three or four mutants per revenue-critical journey will catch the most common form of silent degradation, which is a repair that leaves the test executing the flow but no longer checking its outcome.

Semantic diffing. Syntactic diff size is close to useless as a risk indicator, as the earlier examples showed. - status == "paid" / + status == "authorized" is a two-word change describing entirely different positions in a payment lifecycle: one means funds have moved, the other means a hold exists that may still fail. Conversely - "Delete account" / + "Close account" may be the same operation renamed. What matters is whether the terms occupy the same position in the product's domain model, and that is a question about the domain, not about the strings. This is a place where a language model is unusually useful — asked whether two labels or two status values plausibly denote the same operation, and to explain why, a model produces a reasoned classification that a human can check in seconds. It is also a place to be disciplined about what the output is: a semantic classification is evidence, not authorization. It belongs in the repair record next to the confidence score, as an input to a policy decision that some other component makes.

Automated review of automated repairs. A model can do real work reviewing another model's patch: explaining what the diff changes in behavioral terms, identifying weakened assertions, checking a proposed expectation against a schema, and summarizing evidence for a human. That compresses review time, which is what makes review-required categories affordable. What it does not do is create independence. Two instances of the same model family, given the same context, share failure modes, and a reviewer that inherits the repairer's framing tends to inherit its conclusion. Independence comes from differences in input and objective — a reviewer that reads the requirements and the diff without the repairer's justification narrative, and is asked to argue against the change rather than check it, is a meaningfully different check. Automated reviewers reduce the cost of human review; they do not remove the need for it where being wrong is expensive.

Loud failure edits. One class of change deserves an explicit rule: modifications that make a test stop checking must never be applied quietly. Removing an assertion, skipping a test, marking it flaky, wrapping a failing step in a try/catch, converting a hard assertion to a soft one, or broadening a condition are all rational things to do while debugging.

// A defensible debugging step. An indefensible permanent repair.
try {
  await expect(page.getByTestId('order-total')).toHaveText('$199.00');
} catch {
  test.info().annotations.push({ type: 'known-issue', description: 'total mismatch' });
}

The problem is not that an agent produces this. It is that it can survive review as "test fixed" and then persist for years. Any edit in this class should require human approval, produce a tracked issue, and — the part teams usually skip — carry an expiry, so that a quarantine which is still in place in ninety days becomes a failure of its own rather than a permanent feature of the suite.

One suite, nine changes: a checkout regression example

Abstract policy is easy to agree with and hard to apply. This section works through a single ordinary suite against a single release, because the interesting result is how differently the same policy behaves across changes that all look, at the level of a diff, like the same kind of thing.

The requirement being verified: a signed-in customer with a valid cart can review the order total, select a shipping method, pay with a saved card, submit the order, and receive a confirmed order number.

The suite covers roughly a dozen behaviors: the cart total matches the sum of line items plus tax and shipping; a saved card can be selected; a declined card produces a specific error and does not create an order; shipping method selection updates the total; a loyalty discount applies at the correct rate for eligible customers; the submit control is a keyboard-reachable button with an accessible name; submission issues exactly one order-creation request; a duplicate submit does not create a second order; the confirmation page displays an order number matching the created order; the order reaches confirmed status in the order service; the confirmation email request is dispatched; and an out-of-stock item blocks submission with a specific message.

The release under test changes nine things.

A — A generated element identifier changed. #checkout-btn-8821 became #checkout-btn-9134 after a build-tool upgrade. The failure is deterministic, at the locate step. The evidence is unambiguous: same role, same accessible name, same position, same handler, no network or state difference. Category: structural locator. Semantic impact: none. Apply automatically. This is the case automatic repair exists for, and requiring review here is how teams end up disabling repair entirely.

B — The submit button label changed from "Place order" to "Complete purchase." Structurally identical, still a button, still in the same form, still issuing POST /api/orders. The healer is highly confident. But the accessible name is part of what the test observed, and label changes sometimes accompany behavior changes — a "Complete purchase" that now charges immediately where "Place order" previously created a pending order is a different product. Category: accessible-name change. The corroboration check decides it: if the string appears in a localization file modified by a reviewed pull request, and the differential run shows the same request issued with the same payload and the same resulting order status, propose with evidence for a fast approval. If the label change came with any difference in the request or the resulting state, it is a finding, and the label was the smallest part of what changed.

C — An upsell interstitial was inserted between checkout and payment. Repair is mechanically trivial: click "Continue to payment." Category: navigation step added. The questions the DOM cannot answer are whether this was intentional, whether it blocks users, whether it appears for all customers or an experiment cohort, and whether the funnel has an owner who agreed to it. Propose the patch, but attach a journey-change annotation and route it to whoever owns checkout conversion. Note the second-order effect worth flagging in the record: the suite's duplicate-submission test now traverses an extra page, which may change what it is actually exercising.

D — The confirmation page now displays "Processing" instead of "Order confirmed." This is the exhibit that separates careful teams from fast ones. It looks like copy. The differential run answers it: query the order service directly. If the order reaches confirmed and only the rendered label changed, it is presentational and the corroboration path from B applies. If the order sits in processing because order confirmation moved to an asynchronous worker, then the product's behavior changed materially — the customer no longer receives a confirmed order at the end of checkout, which is the requirement as written. Reject the text repair; raise a finding. The right resolution is a product decision about whether the requirement changed, followed by a deliberate rewrite of the test, possibly splitting it into synchronous acknowledgment and eventual confirmation.

E — The expected total changed from $199.00 to $219.00. Category: expected value. Policy is unconditional here regardless of confidence, and the reason is the one established earlier: the application cannot supply evidence about what it should charge. Corroboration can be searched — a pricing configuration diff, a tax-rule change, an approved promotion expiry — and if found, it goes into the finding to accelerate human resolution. It does not authorize the edit. Reject; raise a finding with the corroboration attached.

F — Payment processing now takes roughly three times longer. The confirmation assertion times out at 5 seconds; the element appears at around 14. The available repair is a larger timeout, and it works. It also encodes a 14-second payment confirmation as acceptable without anyone deciding that. Category: tolerance increase on a revenue-critical test. Reject the silent increase; raise a finding with the measured latency distribution across runs. If the slowdown is accepted, the resolution is an explicit latency assertion and a deliberately updated timeout, not a quietly widened one.

G — The submit control changed from <button> to a clickable <div> with tabindex="-1". Covered in detail earlier. The proposed repair is a semantic downgrade of the locator, and it would pass. Reject; raise a finding. The relevant detail for this section is what the record must say: confidence 0.97, decision reject, rationale that the loss of accessible semantics is the defect. Also worth noting is that the suite's explicit accessibility test — "submit control is a keyboard-reachable button with an accessible name" — fails too, which is the corroborating signal that makes this classification easy. Suites without such a test rely entirely on the locator to carry that information, which is exactly why locator repair is more dangerous in those suites.

H — The seeded loyalty customer no longer exists. If the fixture is requested by identifier, no safe repair exists: nothing in the environment reveals which properties mattered. If the fixture is requested by constraints — premium tier, EU billing, subscription older than 90 days — then substitution is verifiable, and the repair record can carry the proof that the replacement satisfies every declared property. Apply automatically in the constrained case; reject in the identifier case and raise a fixture-provisioning finding. The difference between those two outcomes is a decision the team made months earlier about how to write fixtures, which is a fair summary of how most of this works.

I — A flaky assertion was removed to stabilize the suite. The duplicate-submission test intermittently observed two order-creation requests. The proposed repair deletes the request-count assertion. Category: assertion removal. Reject, unconditionally. The intermittency is the finding: an occasional duplicate order is a real defect with direct financial consequence, and its intermittent presentation is characteristic of a race between the client's disable-on-submit logic and the request. The correct action is evidence collection across runs and a defect report, not a stabilized test.

The pattern across the nine: two applied automatically, three proposed with evidence for review, four rejected as findings. The rejections are not failures of the healer. Four genuine product issues — an unreviewed funnel change, an asynchronous confirmation that broke a requirement, a threefold latency regression, an accessibility regression, and a duplicate-order race — were surfaced rather than absorbed. A permissive healer would have delivered a green suite and none of that information, and every operational metric would have looked better.

Figure 8 — Nine changes, one policy. What it shows: the nine changes as rows, with columns for failure signature, proposed repair, category, semantic impact, and outcome (auto / propose / finding), color-coded by outcome. Where it belongs: closing the checkout example. Caption: One release, one policy, three different outcomes. Uniform treatment of failures is what produces uniform-looking suites and non-uniform product risk.

A review template for a proposed repair

Most of the judgment above compresses into a short list of questions. The value of writing them down is that they can be answered by the system for the easy cases and by a human for the hard ones, in the same format either way.

1. Failure. What exactly failed — which step, which locator, which assertion, with what error? Deterministic or intermittent across runs?
2. Intent. What behavior was this test written to prove? If nobody can answer this in one sentence, that is the finding.
3. Proposed modification. The precise diff.
4. Category. Locator (structural / semantic direction), interaction, synchronization (correction or tolerance), navigation, test data, expected content, assertion, baseline, skip.
5. Evidence for drift. What supports the conclusion that the test is stale rather than the product broken? Name the artifacts.
6. Corroboration. Does any source outside the application under test record this change? Which one, and is it recent enough to trust?
7. Semantic impact. Can user-observable behavior have changed? Did role, accessible name, request payload, resulting state, or journey change?
8. Detection strength. Will the repaired test still catch what the original caught? Were assertions removed, widened, or softened? Were tolerances raised?
9. Criticality. What is the cost if this test becomes less sensitive? Which failure classes stop being observed?
10. Authority. Does policy permit automation to approve this category for this test?
11. Review. If not, who decides, and by when?
12. Rollback. Is the change isolated enough to revert without untangling it from unrelated edits?

If questions 5 through 8 cannot be answered from the collected evidence, the honest output is a finding rather than a patch.

Where automatic repair clearly earns its keep

None of the preceding is an argument for manual maintenance. Reviewing every repair would eliminate most of the benefit and, predictably, lead a team to disable repair entirely after the third week of rubber-stamping identifier churn.

Automatic repair is high-value, low-risk, and worth enabling without hesitation in a well-defined set of cases: generated identifiers and machine-produced class names that change on every build; elements that moved within an equivalent container without changing role or name; replacing a fragile structural query with a role-and-name query; mechanical interaction fixes such as scrolling into view or waiting for the enabled state; replacing fixed sleeps with deterministic waits; framework and API migrations applied uniformly across a suite; and constrained fixture substitution where the replacement demonstrably satisfies declared properties. In a large suite these categories are the overwhelming majority of failures by count, which is the point: restricting automatic repair to high-confidence, low-semantic-impact changes removes most of the maintenance cost while leaving the test's purpose intact. The scarce resource being protected is human attention, and spending it on identifier churn guarantees there is none left for the four findings in the checkout example.

The mirror image — when a healer should stop and surface evidence rather than proceed — follows from the same reasoning. Stop when an assertion or expected business value would change; when permissions, authentication, or destructive-action behavior is involved; when a step in a user journey disappeared; when accessibility semantics were lost; when latency moved outside a meaningful tolerance; when a fixture's scenario properties cannot be verified; when no strong replacement candidate exists; when several plausible candidates disagree, since ambiguity is itself information about the change; and when no source of truth can be established for a change that requires one.

That list has exceptions, and pretending otherwise makes it easy to ignore. A pure localization update to a non-semantic string, corroborated by a reviewed catalog change, is fine to apply. A step that disappeared because an approved simplification removed it is fine once corroborated. An assertion on a nondeterministic identifier was wrong to begin with and should be repaired. The list describes where the burden of proof shifts, not where reasoning stops.

Measuring the right thing, and keeping the ability to undo

A healing rate is not a quality measure. "92% of broken tests repaired automatically" describes throughput and says nothing about whether those repairs were correct — a system that rewrites every failing assertion to match observed behavior would score close to 100%.

Better dimensions, none of which come with universal target values: the proportion of applied repairs later reverted by a human; repairs that touched code paths implicated in an escaped defect within some subsequent window; human rejection rate on proposed repairs, which is a direct read on whether the classifier is calibrated; the distribution of repairs across risk categories, where a rising share of expectation-level repairs is a warning independent of their individual correctness; measured change in detection strength, via assertion counts, predicate specificity, and surviving mutants; the recurrence of failure classes, since the same category of failure repairing itself weekly is a product or infrastructure signal being consumed as maintenance; and, on the positive side, engineering time saved on the low-risk categories, which is the actual benefit and deserves to be counted.

The pairing that matters most is repair volume against detection strength. If both move in a healthy direction, the system is working. If repairs rise while detection falls, the suite is being optimized toward silence, and that will be visible months before it is visible in production.

Whatever the measures, the operational requirement underneath them is reversibility. Automatically applied changes should be isolated — one repair, one commit, one record — rather than batched into a nightly "fix tests" commit that cannot be unpicked. They should carry metadata linking back to the failure and the evidence. They should be visible in review, whether as pull requests or as a queue someone actually reads. And there should be a route to revert a repair without reverting a week of unrelated work. Different organizations will implement this differently, and the shape matters less than the principle: autonomy is safe in proportion to how observable and reversible its changes are.

A structural point supports this. Test code and repair policy should live in different places and be owned differently. The test file states what is being verified and how; the policy states what automation may modify, under what evidence, with what escalation. Burying repair behavior inside opaque infrastructure — a runtime locator-substitution layer that silently resolves a different element than the test names — produces the worst version of this problem, because the test file no longer describes what ran. If a test can pass while interacting with an element it does not name, the source code has stopped being the record.

Test code is release infrastructure

The organizational argument is simple and usually arrives too late. Test suites are not developer convenience. They gate deployments, they determine which defects are discovered before customers find them, they underpin the confidence that allows small teams to ship frequently, and in regulated contexts they are the evidence submitted to auditors. A system with unrestricted authority to modify them has, functionally, unrestricted authority over the organization's quality signal.

That does not mean every locator repair deserves a change-advisory board. It means the authority deserves the same deliberate design as any other capability that can alter release decisions: bounded scope, explicit permissions, retained evidence, auditability, and reversibility. Those are ordinary engineering controls, and they are cheap compared to the cost of discovering that the checkout suite has been green for five months because it stopped checking anything.

QAtronic works with engineering teams on exactly this boundary — designing Playwright and cross-framework automation, repair policies, and evidence practices that cut maintenance cost without quietly trading away defect detection. If you are enabling automated repair across a suite that gates releases, the useful conversation is about which categories to automate and what evidence each one must produce, not about how much of the suite can be healed.

The failure the healer should leave red

Return to the two diffs from the beginning.

- await page.getByRole('button', { name: 'Save' }).click();
+ await page.getByRole('button', { name: 'Save changes' }).click();

- await expect(page.getByTestId('order-total')).toHaveText('$199.00');
+ await expect(page.getByTestId('order-total')).toHaveText('$219.00');

The distance between them is not confidence, and it is not difficulty. A repair system can be equally certain about both, and producing the second is if anything easier — the value is sitting right there in the DOM. The distance is that one edit adjusts how a test navigates a product that changed its wording, and the other edits the sentence that says what the product owes a customer. The first is a repair. The second is an amendment, and amendments have authors.

Everything in this article reduces to that separation. A test that fails is making a claim: something is not as we agreed it would be. The claim might be about the test, in which case repairing it is correct and doing so automatically is a genuine improvement over an engineer spending Tuesday morning on regenerated CSS class names. The claim might be about the product, in which case the failure is the most valuable output the suite produced that night, and converting it into a patch destroys the finding, the evidence, and the record that there was ever anything to look at.

The uncomfortable property of a repair system is that it faces both cases with the same information and the same incentive. Green is the reward signal. There is always an available edit that produces it. This is why the design question is not how accurate the matcher is but where its permissions end — and why the most sophisticated behavior such a system can exhibit is not a clever repair but a declined one:

I can change this test.
I should not change this test.
The failure contains evidence that requires a human or a product decision.

The capability to modify a test and the authority to redefine what it expects are different things, and conflating them is how a suite becomes a mirror. A healer that maximizes repairs is optimizing the metric it was given. A healer worth trusting is one that occasionally hands back a failure it could have made disappear, with the evidence attached, and lets the red stand.

Sources

• Playwright, Playwright Test Agents — planner, generator, and healer roles, init-agents, seed tests, and the specs/ and tests/ artifact conventions. https://playwright.dev/docs/test-agents
• Playwright, Trace viewer. https://playwright.dev/docs/trace-viewer
• Playwright, Snapshot testing (aria snapshots) — toMatchAriaSnapshot and snapshot updating. https://playwright.dev/docs/aria-snapshots
• Playwright, Visual comparisons. https://playwright.dev/docs/test-snapshots
• Playwright, Release notes — --update-snapshots modes (changed, all, missing, none) and --update-source-method. https://playwright.dev/docs/release-notes
• Playwright, Locators and Accessibility testing. https://playwright.dev/docs/locators · https://playwright.dev/docs/accessibility-testing
• S. R. Choudhary, D. Zhao, H. Versee, A. Orso. "WATER: Web Application TEst Repair." ETSE '11. https://doi.org/10.1145/2002931.2002935
• A. Stocco, R. Yandrapally, A. Mesbah. "Visual Web Test Repair." ESEC/FSE 2018. https://doi.org/10.1145/3236024.3236063
• M. Nass, E. Alégroth, R. Feldt, M. Leotta, F. Ricca. "Similarity-based Web Element Localization for Robust Test Automation." ACM TOSEM 32(3), 2023. https://doi.org/10.1145/3571855
• M. Nass, E. Alégroth, R. Feldt. "Improving Web Element Localization by Using a Large Language Model." STVR 34, 2024. https://doi.org/10.1002/stvr.1893
• M. Hammoudi, G. Rothermel, P. Tonella. "Why Do Record/Replay Tests of Web Applications Break?" ICST 2016.
• E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, S. Yoo. "The Oracle Problem in Software Testing: A Survey." IEEE TSE 41(5), 2015.
• M. Eck, F. Palomba, M. Castelluccio, A. Bacchelli. "Understanding Flaky Tests: The Developer's Perspective." ESEC/FSE 2019.
• Y. Jia, M. Harman. "An Analysis and Survey of the Development of Mutation Testing." IEEE TSE 37(5), 2011.
• W3C, Web Content Accessibility Guidelines (WCAG) 2.2. https://www.w3.org/TR/WCAG22/
• W3C, Accessible Rich Internet Applications (WAI-ARIA) 1.2. https://www.w3.org/TR/wai-aria-1.2/