TSM Shifts Channels Through Time

TSM moves a slice of channels one step forward and back in time, giving a 2D network temporal reach with almost no extra cost.

TSM shifts one channel slice from frame 1 into frame 2 and back, so a 2D convolution at frame 2 sees past, present, and future together

Why Does This Exist?

3D convolutions cost cubic compute and 2D networks see no time at all. Video needs something between: temporal mixing at 2D price.

TSM shifts channels along time before each convolution. It powers video classification, scores action recognition, and stays far cheaper than heavy dual paths like SlowFast.

Think of It Like This

Sliding neighbor notes across desks

Each student slides one page to the left desk and one to the right before working. Everyone still works alone, but each desk now holds past, present, and future notes.

It fails when everyone slides everything. With no pages left on the home desk, nobody remembers the present frame clearly.

How It Actually Works

Let $X_t$ hold frame $t$ channels, where $t$ indexes time. Split channels into three groups: shift forward, shift back, and keep. Then $X_t' = [X_{t-1}^{fwd}, X_{t+1}^{bwd}, X_t^{keep}]$ , where $fwd$ brings the past, $bwd$ brings the future, and $keep$ holds the present. A plain 2D convolution follows.

Worked example

With 8 channels, shift 1 forward and 1 back, keeping 6. Frame 2 receives channel 0 from frame 1 and channel 7 from frame 3. Accuracy moves 0.66 without shift to 0.74 with shift, a gain of 0.08 for zero multiplies and only data movement. Shifting 4 of 8 drops back to 0.69 because the present gets too thin.

Watch Out For

Shifting so much the present disappears

Shifting half the channels starves spatial detail. The symptom is motion gains wiped by worse object scores. Keep shifts near one eighth each way and keep the rest resident.

The Quick Version

TSM permutes channels across time, then runs ordinary 2D convolution.
One eighth forward plus one eighth back balances past, future, and present.
The fixture gains 0.66 to 0.74 with no extra multiplies.
Too much shift thins the present frame and scores fall.
It enables online recognition because only past shifts are needed live.