C3D Learns Space and Time Together

C3D slides small 3D cubes through video, so one network reads shape and motion together instead of averaging frame guesses.

C3D stacks 16 frames into a volume and slides 3 by 3 by 3 kernels through it, keeping motion that frame averaging at 0.55 misses to reach 0.78

Why Does This Exist?

Averaging per-frame votes throws away order. Running looks like standing when frames are shuffled, because the average cannot see direction. Motion lives between frames, not inside one.

C3D treats a clip as a 3D volume with width, height, and time. It builds on video understanding, classifies clips with action recognition, and gives the joint alternative to two-stream networks that split space and motion.

Think of It Like This

A flipbook pressed into a loaf

Stack sixteen flipbook pages and cut a small cube from the loaf. Each cube holds a bit of drawing plus how it moved to the next page.

It stops working when the loaf gets huge. Sixteen full frames per decision costs far more flour than tasting one page at a time.

How It Actually Works

Let a clip be $V$ of shape $16 \times 112 \times 112 \times 3$ , where 16 is frames, 112 is height and width, and 3 is colour. A 3D kernel $K$ of shape $3 \times 3 \times 3$ slides in all three axes, where each position sums $3 \times 3 \times 3 \times C$ products. Pooling halves space and time together until one vector remains.

Worked example

A 16-frame clip at 112 pixels needs about 33 million multiplies in the first layer alone. Frame averaging scores 0.55 on a running clip because direction is lost. The C3D volume keeps the leg swing across frames and scores 0.78. The 0.23 gap is pure temporal order, bought with cubic compute.

Watch Out For

Feeding too few frames to a 3D stack

Four frames through three pooling stages collapse time to one before motion forms. The symptom is C3D matching a 2D CNN while costing far more. Feed the full 16-frame clip the kernels were designed around.

The Quick Version

C3D reads a 16-frame volume with $3 \times 3 \times 3$ kernels, not single frames.
Shape $16 \times 112 \times 112 \times 3$ keeps temporal order end to end.
The fixture lifts running accuracy from 0.55 averaged to 0.78 volumetric.
The gain is motion direction, and the price is cubic compute.
Short clips waste the 3D stack, so keep the full temporal window.