Something Something Motion

Hand-object clips where the motion phrase matters, not the object, forcing models to learn temporal direction.

Something Something maps each hand-object clip to a motion phrase, so direction across time decides the label

Why Does This Exist?

Many actions can be guessed from one frame plus the object name. Pushing versus pulling the same cup looks identical still. Something-Something labels clips by motion templates with objects slotted in, so video models must read direction and order rather than scenery.

This page covers motion-phrase contract and appropriate use. Clip backbones are in SlowFast.

Think of It Like This

Same cup, opposite verbs

Hold a cup. Push it away versus pull it closer. The photo matches, the verb flips.

That verb flip is the dataset. Objects stay, directions decide.

How It Actually Works

Purpose is fine-grained temporal classification. Input is a hand-object clip, output is one motion template. Appropriate use is testing temporal reasoning and order sensitivity. Object bias checks matter: swapping objects should not flip a motion prediction.

Toy direction walkthrough

Take 10 push-versus-pull pairs where frame-only gets 5 right by chance. A temporal model getting 8 right gives lift $(8 - 5) / 10 = 0.3$ . The 0.3 is pure order signal on this toy slice.

Code

Order-sensitivity check by reversing time.

1import numpy as np2
3
4def direction_score(forward: np.ndarray, backward: np.ndarray) -> float:5    return round(float((forward != backward).mean()), 3)  # flipped verdicts share6
7
8print(direction_score(np.array([1, 1, 0, 1]), np.array([0, 1, 1, 1])))9# -> 0.5

Half the verdicts flip on reversal, showing time mattered.

Watch Out For

Letting object names leak the answer

Template slots still carry object words, so models can cheat by object. The symptom is strong accuracy that survives frame shuffling. Fix it by testing shuffled frames and unseen objects.

The Quick Version

Something-Something maps hand-object clips to motion phrases where direction decides.
Input is a clip, output is one temporal template.
Toy 8 versus 5 on 10 gives 0.3 order lift, with reversal flips near 0.5.
Use it to test temporal reasoning beyond scenery.
Check shuffles and unseen objects for cheating.