Product Journey Integrity Benchmark v1

The Product Journey Integrity Benchmark v1 publishes actual repeated-browser observations for one outcome across three public open-source TodoMVC implementations: React, Vue, and JavaScript ES6. It measures outcome completion, declared browser-state control, exact visual repeatability, three sampled accessibility semantics, and evidence completeness.

It does not rank frameworks, vendors, or products. The sample deliberately holds the application specification constant so the first version can test the harness and evidence model.

Direct result

Target	Outcome	Browser state	Visual	Sampled accessibility	Evidence
TodoMVC React	Observed failure in 0/3 repetitions: the named primary textbox was absent	Browser context controlled; server state unavailable	Missing because outcome setup failed	Sampled checks failed at the missing textbox	Incomplete
TodoMVC Vue	Observed pass in 3/3 repetitions	Browser context controlled; server state unavailable	Exact screenshot SHA-256 in 3/3	Three sampled checks passed	Complete for method
TodoMVC JavaScript ES6	Observed pass in 3/3 repetitions	Browser context controlled; server state unavailable	Exact screenshot SHA-256 in 3/3	Three sampled checks passed	Complete for method

The React URL returned HTTP 200 in all repetitions but did not expose the named TodoMVC input during the 30-second locator window. The benchmark records this as observed failure, not as proof that React or the source implementation is defective. Hosted demo deployment, asset loading, or current public state may explain the observation.

Dataset and explorer

• Raw dataset
• Interactive results explorer
• Executable harness at scripts/seo/run-product-journey-integrity-benchmark.mjs
• Target source: TodoMVC repository
• Target license: MIT

The dataset is CC BY 4.0; target screenshots and code remain subject to the TodoMVC MIT license.

Journey under test

Every repetition starts in a fresh browser context and attempts the same outcome:

Create the todo “Verify release truth” and observe the named item plus one active item.

The harness verifies HTTP 200, locates a textbox with accessible name “What needs to be done?”, fills and submits the item, waits for exact visible text, and checks the “1 item left” consequence.

Declared browser state

Each repetition declares:

• fresh browser context;
• Chromium through Playwright 1.57.0;
• viewport 1200×748;
• device scale factor 1;
• locale en-US;
• light color scheme;
• reduced motion;
• disabled screenshot animations and hidden caret.

The benchmark does not reconstruct server identity, account, database fixtures, feature flags, dependencies, clock, or side effects. State status is therefore browser-state-controlled, not complete State Capsule verification.

Visual method

After the outcome passes, the harness captures a full-page PNG. Exact SHA-256 equality across three repetitions is reported as exact-within-run. It also hashes the final HTML DOM.

Exact repeatability is limited to one machine, Chromium build, network period, viewport, and host state. It does not prove cross-browser or cross-runner determinism. A matching screenshot does not prove the outcome or accessibility by itself.

Sampled accessibility evidence

The harness checks:

1. one named primary textbox;
2. one level-one heading named “todos”;
3. at least one exposed checkbox after the item is created.

These checks test the semantic targets required for this Journey. They are not a WCAG audit, manual protocol, assistive-technology evaluation, or conformance statement. The dataset labels them sampled-checks-pass or sampled-checks-fail accordingly.

Evidence retained

For every repetition the dataset records:

• target URL and source repository;
• start and completion timestamps;
• HTTP response status;
• outcome assertion IDs, values, and pass/fail;
• accessibility check IDs, values, and pass/fail;
• screenshot URL and SHA-256 when captured;
• DOM SHA-256 when captured;
• exact error text when execution fails;
• runtime and declared state.

This makes failures inspectable instead of reducing the benchmark to a score.

Why no composite score

Outcome, state, visual, accessibility, and evidence answer different questions. An exact screenshot cannot compensate for a missing outcome. A passing sampled accessibility check cannot establish conformance. A complete evidence record can faithfully describe failure.

The explorer keeps dimensions separate.

Source and licensing controls

Every target record includes the public URL, exact framework label, source-repository path, MIT license URL, and observation timestamps. The harness does not copy or execute repository code locally in v1; it observes the current public deployment. That means the hosted bytes may differ from the repository’s latest branch.

Screenshots are evidence of the observed public state and should retain target attribution. Dataset commentary is licensed CC BY 4.0. The source applications retain MIT licensing. Future targets require a documented license and redistribution review before raw captures are published.

Correction policy

If a target maintainer identifies an error, provide the relevant target, repetition, source URL, and contradictory evidence. A correction updates the current benchmark page and creates a new dataset version when observations change. The original August 24 dataset remains available as historical evidence.

A later successful React run would not make the original recorded timeouts false. It would show that public state changed or the run conditions differed.

Independent evaluation

The harness produces deterministic observations. A separate reviewer should verify that the dataset matches the captured artifacts, the prose does not overstate results, and every limitation remains visible. The same LLM that drafts the explanation cannot be the sole approver.

Reproduce

Install the project’s pinned Chromium and run:

pnpm --dir app exec playwright install chromium
node scripts/seo/run-product-journey-integrity-benchmark.mjs

The command writes screenshots and dataset.json beneath app/public/research/product-journey-integrity-benchmark-v1/. Public hosted targets can change; a new run creates new observations rather than retroactively altering what the August 24 dataset observed.

Limitations

• Three TodoMVC implementations are not representative of SaaS complexity.
• All targets share the same application specification and public host.
• The test covers one client-side outcome and no authentication or server side effects.
• Network and host deployment are outside benchmark control.
• Accessibility evidence covers only three semantics.
• Exact visual/DOM repeatability covers one run and machine.
• The React failure identifies the observed public surface, not root cause.
• No customer, conversion, productivity, or defect-detection outcome is measured.

Next version

Version 2 should add preregistered open-source SaaS applications pinned to source commits and reproducible local containers. It should include authenticated or seeded state, outcome-specific accessibility protocols, multiple browser engines, and signed evidence packages while preserving the original stable dataset URL and a versioned archive.

Expansion must not occur until collection, licensing, source hashing, and independent review are reproducible.

Read the audience interpretations for engineering and QA, documentation and support, and product marketing.