Reproducibility in Vision Research

Published gains vanish on rerun when seeds, data versions, and missing code decide results more than methods. Reproducibility makes vision claims checkable.

Paired seeds show plus 0.3 three times running, a consistent gain no single lucky seed could prove

Why Does This Exist?

Vision papers report headline seeds: the best of many runs presented as the method's worth. Reruns with new dice land lower; "state of the art" dissolves into seed selection. Missing code, unversioned datasets, and environment drift finish the job — independent teams cannot even reproduce the setup, let alone the number. Trust in reported deltas erodes with every quiet failure.

Reproducibility standardizes checkability: multiple seeds, shared code and environments, pinned data versions, and statistical comparisons that separate signal from dice. The seed-noise warning is open research; the test-design half is benchmark design.

The one idea you need first. A paired-seed comparison runs baseline and method on identical seeds and tests the differences: same dice, both runners, fair race. Three wins of $+0.3$ across paired seeds evidence a real (if small) effect; one unpaired $+1.0$ evidences a lucky draw. Pairing removes seed luck from the comparison the way double-blind removes examiner bias.

Think of It Like This

A recipe that only works in the author's kitchen

A cookbook recipe flops in your kitchen: "a hot oven" (how hot?), "a pinch" (how much?), unlisted altitude, a grandmother's seasoned pan doing invisible work. The dish existed once, under unrecorded conditions, and no reader can return there.

Irreproducible papers are that recipe: headline numbers baked in unlisted ovens (seed $42$ , forked data, library versions of March). Reproducibility writes grams and degrees: seeds listed, code posted, data versioned, environment pinned. Readers' kitchens differ, but now they can calibrate — and now they can cook.

How It Actually Works

Checklists gate publication: three or more seeds with means and spreads, code plus environment (containers, lockfiles), dataset version pins, and ablations isolating the claimed cause. Comparisons use paired tests over seeds (paired t-tests, Wilcoxon, or bootstrap intervals) rather than headline-vs-headline. Nondeterminism sources get documented: GPU atomics, dataloader orders, augmentation dice — each either seeded or disclosed.

Artifact review (code that runs, data that downloads) increasingly accompanies paper review; badges mark the verified.

Worked example

Baseline across seeds $[80.9,\ 80.1,\ 81.5]$ : mean $(80.9+80.1+81.5)/3 = 242.5/3 \approx 80.83$ . Method on the same seeds $[81.2,\ 80.4,\ 81.8]$ : mean $243.4/3 \approx 81.13$ . Paired differences: $81.2-80.9 = 0.3$ , $80.4-80.1 = 0.3$ , $81.8-81.5 = 0.3$ .

Three $+0.3$ s, all positive, mean gain $+0.3$ with zero spread in the difference: small, consistent, believable — the opposite of one unpaired $+1.0$ that seed noise could have dealt. Consistency across pairs is the evidence; the headline gap alone never was.

Code

The point of this snippet is the paired verdict: differences per seed, not headlines per method.

1def paired_gain(baseline: list, method: list) -> tuple:2    diffs = [m - b for m, b in zip(method, baseline)]3    mean = sum(diffs) / len(diffs)4    return [round(d, 2) for d in diffs], round(mean, 2)5
6
7print(paired_gain([80.9, 80.1, 81.5], [81.2, 80.4, 81.8]))  # -> ([0.3, 0.3, 0.3], 0.3)

Watch Out For

Code without the data it ran on

Repositories post training scripts against datasets since removed, relicensed, or silently updated — the code runs, the numbers never return. Gated medical and web-scale sets rot fastest; links die within years.

Pin dataset versions with hashes, vendor a runnable subset, and document access paths for the rest. Reproducibility that needs unobtainable inputs is documentation, not science.

Nondeterminism hiding in the stack

GPU reductions, dataloader parallelism, and library updates inject unseeded randomness: identical code and seeds still wobble across machines and months. Teams chasing bit-exactness burn quarters; teams ignoring it mistake stack wobble for method effects.

Seed everything seedable, record the environment exactly, and size claimed gains above the measured stack wobble. Bit-exactness is a bonus; wobble-aware statistics are the requirement.

The Quick Version

Reproducibility needs seeds, code, environments, data versions, and paired statistics together.
The worked fixture shows paired gains $[0.3, 0.3, 0.3]$ : small, consistent, believable.
Pair baseline and method on identical seeds; headline-vs-headline proves nothing.
Vendor runnable data subsets and hash full versions, or links rot the claim.
Size gains above measured stack wobble; bit-exactness is bonus, statistics are required.