Temporal Grounding

Finding when something happened matters as much as what. Temporal grounding maps language to timestamps: the exact seconds where the query lives in the video.

Prediction overlaps truth across 15 of 25 seconds for IoU 0.6, clearing the 0.5 bar for a counted hit

Why Does This Exist?

Hours of footage hold seconds of interest: the goal in the match, the defect on the line, the moment the suspect appears. Scrubbing wastes expert hours; whole-video classifiers answer "yes it happened" without saying when. Products need timestamps — jump links, evidence clips, alert windows — not bag-level verdicts.

Temporal grounding (moment retrieval) takes a video plus a language query and returns start-end spans. Architectures split into proposers (enumerate candidate spans, rank by query match) and span predictors (regress start/end distributions directly, QA-style). The metric is temporal IoU against human marks. The narrative home is long-form understanding; the clip classifier is action recognition.

The one idea you need first. Temporal IoU mirrors box IoU in one dimension: intersection seconds over union seconds of predicted and true spans. Recall@ $1$ at IoU $\ge 0.5$ asks whether the top prediction overlaps truth by at least half — the standard hit criterion. Boundary fuzz (humans disagree by seconds) motivates looser thresholds ( $0.3$ ) alongside strict ones ( $0.7$ ).

Think of It Like This

Finding one quote in a podcast

Nobody re-listens to three hours for one sentence. Transcript search jumps to candidate paragraphs; surrounding context confirms the exact quote; the share link carries the timestamp. Skim, verify, cite — three moves, seconds total.

Grounding pipelines mirror the ritual. Proposals are transcript hits (candidate spans); query matching is the context read (does this span show the goal?); the span output is the timestamp link. Whole-video classification is the friend who confirms "yeah it was in there somewhere" — technically right, practically useless.

How It Actually Works

Video and query encode jointly (cross-attention between frames and words), then heads predict: start/end probability distributions over time (span style, trained with KL against Gaussian-smoothed truth boundaries) or proposal scores over sliding windows and scales. Non-maximum suppression dedupes overlapping predictions. Training softens boundaries (label smoothing around human marks) because exact frames are annotator fiction. Inference thresholds or takes top- $k$ spans per query.

Scale handling decides quality: multi-scale feature pyramids catch $2$ -second gestures and $2$ -minute procedures in one pass.

Worked example

Truth: the goal spans $[30, 50]$ s. Prediction: $[35, 55]$ s. Intersection: $[35, 50] = 15$ s. Union: $[30, 55] = 25$ s. IoU $= 15/25 = 0.6$ .

At threshold $0.5$ : hit (counts toward Recall@ $1$ ). At threshold $0.7$ : miss. One prediction, two verdicts — which is why papers report $0.3/0.5/0.7$ curves instead of single numbers, and why a "state of the art $0.62$ " needs its threshold attached to mean anything.

Code

The point of this snippet is the verdict: intersection over union decides hit or miss at each threshold.

1def temporal_iou(pred: tuple, truth: tuple) -> float:2    inter = max(0.0, min(pred[1], truth[1]) - max(pred[0], truth[0]))3    union = max(pred[1], truth[1]) - min(pred[0], truth[0])4    return inter / union  # 1-D overlap ratio5
6
7iou = temporal_iou((35.0, 55.0), (30.0, 50.0))8print(round(iou, 2))  # -> 0.69print(iou >= 0.5, iou >= 0.7)  # -> True False

Watch Out For

Boundaries humans never agreed on

Annotators marking "the goal" differ by seconds (run-up included or not?), so the $0.5$ threshold forgives what $0.7$ punishes arbitrarily. Models tuned to one dataset's boundary conventions transfer poorly where conventions differ.

Smooth training labels around marks and report multi-threshold curves. A model winning only at $0.3$ retrieves neighborhoods, not moments — fine for browsing, fatal for evidence.

Proposals biased to middles and longs

Sliding windows oversample middle-length spans in the video center; datasets overrepresent medium moments. Models learn the prior (predict central, medium spans) and score respectably while grounding nothing — the temporal version of language bias in VQA.

Debias evaluation with out-of-distribution splits (odd positions, odd lengths) and check prediction histograms against uniform. A grounding model predicting the dataset mean is a prior with extra steps.

The Quick Version

Temporal grounding returns start-end spans for language queries over video.
The worked fixture gives IoU $15/25 = 0.6$ : a hit at $0.5$ , a miss at $0.7$ .
Report multi-threshold curves; single numbers without thresholds mislead.
Smooth boundary labels during training because exact frames are annotator fiction.
Debias with odd-position splits, or models learn to predict the dataset's middle.