Diagnose End-to-End Test Failures

To Diagnose End-to-End Test Failures, preserve the first attempt and classify evidence before retrying, healing a target, increasing a timeout, or accepting a baseline.

Functional/product failure

Actions completed but the observable outcome is wrong, missing, duplicated, or applied to the wrong entity. Block the affected cell and inspect product evidence. Do not retry into a false green.

State failure

Actor, role, account, plan, data, flags, locale, clock, dependencies, or side-effect policy differ from the State Capsule. Repair state or correct scope. Authentication success does not prove correct authority.

Target failure

The intended control is missing, ambiguous, stale, or unsafe. Repair the target contract. A self-healing locator must not select a semantically different action without Review.

Visual change

Candidate and baseline differ inside a compatible context. Inspect side-by-side, overlay, heatmap, DOM, accessibility-tree, and history. Decide intended change, defect, or nondeterminism before updating the baseline.

Accessibility barrier

Automated or manual evidence identifies a participation barrier. Assign owner, remediation, verification, or a scoped expiring exception. Do not suppress the rule permanently.

Privacy or policy failure

Evidence contains prohibited data, unsafe side effect, or policy violation. Quarantine the artifact and block Publication. Passing functional assertions do not override it.

Infrastructure exception

Browser crash, runner disconnect, unavailable local service, or storage/network infrastructure can retry under bounded policy. Preserve every attempt and mark exhaustion separately.

Evidence packet

Retain release/build identity, Journey Version, context, runner, attempt, functional assertions, trace, screenshot/DOM/accessibility evidence, console/network results, state/target diagnostics, error, and timestamps. Redact private data before public use.

Decision

Route exact subjects to authorized owners. Record rationale and evidence. A framework’s automatic classification can inform Review but cannot become the sole release Decision.

Diagnostic order

1. confirm release/build and source identity;
2. verify State Capsule materialization;
3. inspect target resolution and actionability;
4. evaluate the observable functional checkpoint;
5. inspect visual/accessibility peers;
6. review network, console, privacy, and policy evidence;
7. classify root cause confidence;
8. decide retry, repair, block, accept risk, or request evidence.

This order avoids blaming the product before confirming state and target while still refusing to hide a wrong outcome.

Retry evidence

Record every attempt, runner identity, start/completion time, failure IDs, and outcome. A later pass can classify a test flaky or infrastructure-sensitive, but it does not erase the initial observation. Set bounded maximum attempts.

Healing boundary

A healed target must remain semantically equivalent: same user-facing purpose, authority, and expected consequence. Log the proposal and source change. Require Review when confidence is insufficient or the change alters protected source.

Team handoff

Send product defects to owning engineering/product, state/fixture failures to environment owners, target failures to source maintainers, accessibility barriers to component/accessibility owners, privacy/policy to authorized security/legal owners, and infrastructure to platform operations. Retain one Run identity across the handoff.

Metrics

Track failures by class, first-attempt failure, retry outcome, time to classification, reopened Decisions, repeated weak targets/state, and false-green prevention. Do not optimize only for a lower red-test count.

Use repeated failure clusters to improve State Capsules, target contracts, runner infrastructure, and product ownership. Keep the original evidence and do not rewrite prior classification without an attributed correction.

Anti-patterns

Avoid arbitrary sleeps, unlimited retries, automatic baseline replacement, selector broadening, swallowed console/network failure, defaulting missing evidence to pass, and writing “flaky” without a retained diagnostic.

First practice

Create one controlled example of product failure, state mismatch, target failure, and infrastructure exception. Verify only infrastructure retries automatically and every result remains distinguishable in the Run and Release Book.

Continue with Product Journey Runs, Stateful End-to-End Testing, and Functional, Visual, and Accessibility Evidence in One Run.