Open Research Problems

Why Does This Exist?

Benchmark curves flatten near $99\%$ while deployed systems keep failing: novel poses, rare conditions, confident hallucinations, irreproducible gains. The leaderboard measures a solved-looking slice; the street serves the unsolved remainder. Without a map of what is actually open, the field optimizes decimals on closed problems.

Open research organizes the remainder into four frontiers. Generality: one model for many tasks (unified models, generalist agents). Efficiency: eyes for labs without GPU-years (efficient pretraining, data-centric AI). Honesty: tests that resist gaming (benchmark design, hallucination). Grounding: scenes with time and physics (4D understanding, physically grounded generation) — all held together by reproducibility.

The one idea you need first. Seed noise is the spread of results across random training seeds: same code, same data, different dice. A claimed gain smaller than that spread is indistinguishable from luck. Reporting means plus ranges over three or more seeds is the minimum honest unit — everything below is anecdote with error bars missing.

Think of It Like This

How It Actually Works

Each frontier has a program. Generality builds shared backbones with task routing and instruction following, graded on breadth (worst-task scores, not averages). Efficiency buys eyes cheaply (masking, filtering, staging) and spends data quality over quantity. Honesty redesigns evaluation (contamination audits, prior-shifted splits, human-necessity tests) and penalizes fluent falsehood. Grounding adds time, depth, and physics to representations. Reproducibility wraps all four: seeds, code, data versions, and statistical comparisons as entry tickets.

No frontier advances on architecture alone; each is half measurement reform.

Worked example

A paper claims $+0.5$ points from a new module. Reruns across three seeds give $81.2$, $80.4$, $81.8$:

1. Mean: $(81.2 + 80.4 + 81.8)/3 = 243.4/3 \approx 81.13$.
2. Range: $81.8 - 80.4 = 1.4$ points of pure seed luck.
3. Claim $0.5$ sits inside noise spanning $1.4$ — the module's effect is unproven, whatever the headline seed scored.

Honest reporting shows $81.1 \pm$ seed spread with a significance test, or more seeds. The field's replication record suggests many published deltas live exactly here: real-looking, luck-sized.

Code

The point of this snippet is the noise check: mean and range decide whether a gain survives its own dice.

1def seed_verdict(scores: list, claimed_gain: float) -> tuple:2    mean = sum(scores) / len(scores)3    spread = max(scores) - min(scores)4    return round(mean, 2), round(spread, 2), claimed_gain < spread5
6
7print(seed_verdict([81.2, 80.4, 81.8], 0.5))  # -> (81.13, 1.4, True)

A True in the last slot means the claim drowns in noise: run more seeds or retract the delta.

Watch Out For

The Quick Version

• Four frontiers organize the unsolved: generality, efficiency, honesty, grounding — wrapped by reproducibility.
• The worked fixture shows seed spread $1.4$ swallowing a claimed $+0.5$ gain; report means with ranges.
• Each frontier is half measurement reform: new tests and standards, not just new architectures.
• Saturated decimals transfer nowhere; fund dragon-sounding over harbor re-measuring.
• Version every dataset and metric, and re-baseline when versions change.

What to Read Next

• Unified Vision Models
• Reproducibility in Vision Research
• Hallucination in Vision-Language Models