RAFT Recurrent Flow Refinement

Look up motion, fix a little, repeat. RAFT keeps one high-resolution flow field and polishes it with a learned recurrent step.

One flow field loops through lookups and GRU updates until error settles

Why Does This Exist?

Pyramids like PWC-Net freeze coarse mistakes; single-shot FlowNet cannot revise. RAFT exists to refine instead: build one all-pairs correlation volume at high resolution, then apply a learned GRU update many times on a single flow estimate. Accuracy rises with iterations while weights stay shared and small.

Global matching without recurrence is GMFlow.

Think of It Like This

Tracing with repeated checks

Sketch a face roughly, then glance at the model and fix the nose, glance again and fix the eyes, repeating until likeness locks. Each glance is cheap; the drawing stays one sheet.

Flow is the sketch and correlation lookups are the glances. The analogy stops at convergence: artists stop by feel while RAFT runs a fixed dozen steps with decaying loss weights.

How It Actually Works

Dual encoders output $1/8$ -resolution features; their outer product forms a $4$ D correlation volume pooled into four scales. Each iteration indexes the volume around the current flow estimate, encodes the retrieved costs with flow and context features, and applies a convolutional GRU: $h_{k+1} = \mathrm{GRU}(h_k, x_k)$ , $\Delta f = \mathrm{head}(h_{k+1})$ , $f_{k+1} = f_k + \Delta f$ . Loss weights later steps exponentially more, so early exploration is free and late precision pays.

Worked numbers

Initial zero flow gives $8.5$ px error on a fast Sintel clip. After $4$ steps error falls to $2.7$ px, after $12$ steps to $1.43$ px, and after $24$ steps to $1.38$ px with diminishing returns. The volume for $64 \times 80$ features holds about $26$ million correlations, pooled cheaply across scales.

Code

1def raft_update(flow: float, delta: float) -> float:2    return flow + delta  # one learned GRU step on the field3
4
5print(raft_update(2.0, 0.7))  # -> 2.7

Watch Out For

Running too few steps on fast motion

Two iterations leave large shifts half-solved with streaky fields. The symptom is clean slow regions beside smeared fast cars. Budget twelve steps for benchmarks and at least six for real time, and never judge RAFT by a single pass.

The Quick Version

RAFT keeps one flow field and refines it recurrently.
All-pairs correlation supplies global match evidence upfront.
A convolutional GRU proposes each additive update from lookups.
Error falls from $8.5$ to near $1.4$ px over twelve steps on fast clips.
Shared weights keep the model small despite many iterations.
Step count is the accuracy knob at test time.