Visual Determinism Benchmark

The Visual Determinism Benchmark is an open calibration harness for measuring screenshot variance inside a declared browser context. Version 1 compares eight repeated captures in each of three contexts: a controlled desktop state, a controlled mobile/localized state, and an intentionally uncontrolled animated state.

The result is narrow but useful. Both controlled cohorts produced exact PNG matches across all seven comparisons with their reference capture. The intentionally animated cohort produced no exact matches and a maximum changed-pixel ratio of approximately 0.0303%. This demonstrates that the harness can distinguish stable captures from a small known source of motion on the machine used for the run.

It does not prove that all applications, runners, browsers, or operating systems will be deterministic.

Results

Generated on August 24, 2026 with Playwright 1.57.0, Chromium 143.0.7499.4, Node 22.23.1, and macOS on arm64. The raw dataset records the exact runtime string produced by the harness.

Declared context	Repetitions	Exact matches against reference	Maximum changed-pixel ratio	Interpretation
Controlled desktop, light, en-US	8	7 of 7	0	Stable within this run
Controlled mobile, dark, fr-FR	8	7 of 7	0	Stable within this run
Desktop with intentionally uncontrolled animation	8	0 of 7	0.0303%	Small animation was reliably detected

Download the raw benchmark dataset. The executable harness is stored at scripts/seo/run-visual-determinism-benchmark.mjs in the Reshot repository.

Method

Each cohort declares viewport, locale, color scheme, reduced-motion preference, screenshot animation policy, and whether volatility is intentional. The harness creates a fresh page for every capture while keeping the browser context configuration constant.

For the controlled cohorts it:

• uses fixed copy and identifiers;
• uses Arial rather than a network-loaded web font;
• disables screenshot animations;
• hides the caret;
• sets a fixed viewport and device scale factor;
• declares locale, color scheme, and reduced motion;
• performs no external network requests;
• captures the full deterministic fixture after load.

The volatile cohort uses the same fixed content but allows a small CSS animation and samples it at different delays.

The first capture in each cohort is the reference. Each later PNG is evaluated by:

1. SHA-256 equality for an exact-match result;
2. raw RGBA pixel comparison;
3. count and ratio of pixels with at least one changed channel;
4. mean absolute channel delta;
5. maximum channel delta.

Comparisons never cross declared contexts. The mobile dark French capture is not compared with the desktop light English capture because those differences are intentional product state, not nondeterminism.

Why exact hashing and pixel comparison are both retained

A PNG hash is a strict reproducibility signal. If any byte changes, the hash changes. Pixel comparison explains whether the decoded image actually changed and by how much. Retaining both prevents a metadata-only PNG difference from being confused with product pixels and makes small moving regions visible in the statistics.

The benchmark reports no pass/fail threshold for production UI. A team should first identify and control variance, then establish review policy based on the purpose of each checkpoint. A permissive threshold can hide a defect as easily as it can suppress antialiasing noise.

What the first run shows

The two controlled contexts were exact within one browser build and machine during one benchmark run. This is evidence that the fixture and harness can produce stable captures under those conditions.

The animated dot changed only a small fraction of the page, yet every comparison hash differed and the pixel detector recorded nonzero variance. A dashboard that reported only a rounded “99.97% match” could make that movement look irrelevant. For a tiny status indicator, however, the changing region may carry product meaning. Review policy needs checkpoint context, not only a global percentage.

Limitations

This v1 result has deliberate boundaries:

• it runs on one arm64 macOS machine;
• it uses one Chromium and Playwright build;
• it measures a controlled fixture rather than the live Reshot application;
• it does not test Linux containers, GPU differences, Firefox, WebKit, headed mode, remote runners, or network-loaded fonts;
• it does not estimate visual-defect detection accuracy;
• it does not compare threshold algorithms or human review performance;
• the volatile cohort is a sensitivity control, not a recommended capture configuration.

No result should be presented as a customer outcome or universal false-positive rate.

Reproduce the dataset

From the repository root, install the pinned Playwright Chromium runtime and run:

pnpm --dir app exec playwright install chromium
node scripts/seo/run-visual-determinism-benchmark.mjs

The command overwrites the versioned public JSON dataset with actual observations and prints cohort summaries. Review the runtime, generated time, method, individual hashes, and pixel comparisons in the output before publication.

Next benchmark versions

Version 2 should execute the same fixed fixture across the supported Linux CI image and local macOS environment, preserving runner image identifiers. Version 3 should add Firefox and WebKit without comparing different engines as though they should share pixels. Later versions can test controlled font loading, device scale, timezone, masking, clock policy, and a real approved Reshot product checkpoint.

Each expansion must publish its environment matrix, raw observations, harness revision, limitations, and material changelog. Older data should remain addressable rather than being silently rewritten into a more favorable result.

Use the Visual Test Readiness Checker to audit a proposed baseline, Playwright Visual Testing at Scale for implementation guidance, and Full-Journey Visual QA to connect diffs to outcome evidence and Review.