Real-Time Vision Constraints

Real time means every frame meets its deadline, so budgets split capture, infer, and track with slack for jitter.

Capture 5 plus infer 18 plus track 4 plus draw 3 totals 30 ms with 3.3 ms slack at 30 fps

Why Does This Exist?

Averages lie on cameras. A detector at 28 fps average still misses a theft window when one frame in a hundred takes 45 ms. Real time constraints exist to name the deadline and the drop rule: 30 fps means 33.3 ms per frame, every frame, or the frame is late and the system must do something explicit.

Robots, gates, and theft alerts pay for misses. A late stop box is a collision. A late gate read is a tailgate. Inference optimization buys headroom. Batching trades latency for throughput and often breaks deadlines. Edge deployment sets where each stage runs.

Basics first: latency is time per frame. Deadline is 1000 divided by fps. Jitter is p99 minus median. Slack is deadline minus pipeline sum. Drop policy says what skips when slack goes negative.

Think of It Like This

Trains that must leave on time

A station that averages on time still strands passengers when one train in a hundred leaves 12 minutes late. Timetables work because each leg gets minutes plus slack, and late trains are held or skipped by rule, not by hope.

Camera pipelines are timetables. Capture, infer, track, and draw each get milliseconds plus 3.3 ms slack. Late frames drop by policy instead of dragging the next frame late too.

How It Actually Works

Split the 33.3 ms budget for 30 fps. Capture 5 ms for MIPI plus ISP. Infer 18 ms INT8 on target silicon. Track 4 ms for IoU plus reidentify. Draw plus encode 3 ms. Sum is 30 ms since 5 plus 18 plus 4 plus 3 equals 30. Slack is 3.3 ms since 33.3 minus 30 equals 3.3. That slack absorbs scheduling jitter, not model growth.

Measure p99, not mean. Collect 10,000 frames on locked clocks in a thermal chamber. If median is 30 ms and p99 is 45 ms, 1 percent miss the 33.3 ms deadline even though the average claims 33 fps. Throttle adds 20 percent in 45 C ceilings, turning 30 ms into 36 ms and erasing slack entirely.

Name the drop rule. Newest only keeps the latest frame and drops queued ones for live alerts. Every Nth processes one in three for counting and sleeps the rest. No queue grows unbounded, since a growing queue is lateness stored for later.

Worked deadline math

Deadline is 33.3 ms since 1000 divided by 30 equals 33.33. Pipeline is 30 ms since 5 plus 18 plus 4 plus 3 equals 30. Slack is 3.3 ms since 33.3 minus 30 equals 3.3. Throttled infer at plus 20 percent is 21.6 ms since 18 times 1.2 equals 21.6, pushing total to 33.6 ms and missing by 0.3 ms.

Code

The point is the deadline check before promising 30 fps. It recomputes budget, slack, and throttled total.

1import numpy as np2
3
4def deadline_ms(fps: float) -> float:5    return 1000.0 / fps  # ms per frame allowed6
7
8def pipeline_ms(stages: np.ndarray) -> float:9    return float(np.sum(stages))  # capture plus infer plus track plus draw10
11
12stages = np.array([5.0, 18.0, 4.0, 3.0])13print(deadline_ms(30.0))  # -> 33.33333333333333614print(pipeline_ms(stages))  # -> 30.015print(deadline_ms(30.0) - pipeline_ms(stages))  # -> 3.33333333333333616print(5.0 + 18.0 * 1.2 + 4.0 + 3.0)  # -> 33.6

Qualify with p99 plus throttle minutes, never with mean fps on a bench. Mean hides the 1 percent that misses.

Watch Out For

Mean fps that hides p99 misses

A report claims 33 fps while 1 percent of frames take 45 ms. The symptom is clean averages with weekly missed events. Fix it by gating releases on p99 under deadline for 10,000 frames and by plotting latency histograms, not single numbers.

Batching that buys throughput with deadlines

Batch 8 lifts GPU queries per second while pushing batch latency to 25 ms plus queue wait. The symptom is higher throughput with late alerts. Fix it by keeping cameras at batch 1 with newest only drops, and by batching only in cloud retrain paths.

The Quick Version

Real time means each frame meets 33.3 ms at 30 fps, not average fps.
Our split totals 30 ms with 3.3 ms slack for jitter.
Throttle adds 20 percent, turning 18 ms infer into 21.6 ms and erasing slack.
Gate on p99 under deadline for 10,000 frames in heat.
Keep cameras at batch 1 with newest only drops when slack goes negative.