Product Journey Integrity for Engineering and QA

Product Journey Integrity for Engineering and QA interprets the v1 benchmark as a harness and evidence result, not a framework ranking. Vue and JavaScript ES6 completed the one-todo outcome in all three repetitions with exact screenshot hashes and passing sampled semantics. React returned HTTP 200 but never exposed the required named textbox, so every repetition failed before evidence capture completed.

HTTP success is not outcome success

The React target demonstrates why status 200 is weak release evidence. The application shell was reachable, but the required interaction contract was absent during the locator window. Engineering should inspect asset delivery, runtime errors, deployment configuration, and current hosted source before calling this a product defect.

The harness correctly preserved failure instead of changing selectors, increasing retries, or choosing another input.

State is only partially controlled

Fresh browser contexts isolate cookies and storage and declare viewport, locale, color scheme, reduced motion, and scale. That is not complete application state. The benchmark does not seed server data, authenticate an actor, fix a clock, control dependencies, or verify cleanup.

For a real SaaS Journey, add a State Capsule and distinguish browser isolation from product-state reconstruction.

Exact screenshots need outcome evidence

Vue and JavaScript ES6 produced identical PNG SHA-256 values across their three repetitions. This shows within-run repeatability for those public states. It does not show cross-browser compatibility, visual-defect sensitivity, or correctness.

Keep the outcome assertions, state, screenshot hash, DOM hash, and error together. Do not promote visual exactness into release approval.

Accessibility evidence is scoped

The checks cover the named primary textbox, level-one “todos” heading, and exposed checkbox. They verify the semantic targets needed for this Journey. They do not test keyboard sequence, focus, announcements, contrast, text alternatives, zoom, manual protocols, or WCAG conformance.

Treat automated semantics as peer evidence and add manual protocols where outcome risk requires them.

Evidence-complete failure is useful

The React result retains HTTP status, failed accessible-target count, timestamps, and exact timeout. Evidence status is incomplete because no screenshot/DOM artifact was captured after outcome completion. That distinction is more actionable than one failed score.

Future harness versions should capture a failure screenshot and DOM hash in catch before closing the context, while keeping the failure classification unchanged.

CI adoption

Use the harness pattern for one release-critical Journey:

1. preregister outcome and selectors by accessible meaning;
2. declare state and unsupported dimensions;
3. run multiple fresh repetitions;
4. preserve initial failures;
5. capture functional, visual, accessibility, state, and diagnostic evidence;
6. classify root cause separately from observed failure;
7. require Review before baseline change;
8. publish raw evidence and limitations for benchmark claims.

Failure-artifact improvement

The current harness captures screenshot and DOM only after the outcome passes. That protects the passing visual comparison from incomplete state but leaves failure diagnosis weaker. Version 2 should capture a separately labeled failure screenshot, DOM, console, and failed requests inside the error branch.

Those artifacts must never enter the passing visual cohort or be interpreted as a baseline. Retain them under a diagnostic evidence type.

Framework neutrality

All three targets share the TodoMVC specification but differ in public deployment state. The harness should treat accessible names and observable outcomes as the contract, not implementation-specific component structure. Avoid framework-specific selectors unless the research question explicitly concerns implementation.

Repeat the run from a pinned local source commit before making any claim about repository behavior.

Do not infer

Do not infer React quality from one hosted demo failure. Do not infer Vue or JavaScript framework superiority from two passing targets. Do not infer no visual bugs from exact screenshots. Do not infer accessibility conformance from three checks.

The valid conclusion is narrower: the harness observed two repeatable successful public outcomes and one repeatable missing-target failure under the declared conditions.

Inspect the raw dataset, use the results explorer, and read Stateful End-to-End Testing.