Physical Reasoning

Blocks fall, balls bounce, towers topple — and toddlers predict all of it. Physical reasoning gives vision models the same intuition: mass, support, and collisions read straight from video.

A thrown ball lands at 8.8 meters by projectile math while a physics-free guess lands at 6, separating dynamics from pattern matching

Why Does This Exist?

Recognition cannot stack boxes. Knowing a tower's pixels says nothing about whether it stands after the next block — that needs support, gravity, and collision reasoning that classifiers never learned. Robots that plan with recognition alone push towers over with tractor confidence.

Physical reasoning studies intuitive physics in machines: permanence (hidden things persist), support (unbalanced things fall), collision (momentum transfers), containment. Tested with violation-of-expectation probes borrowed from infant studies, it separates models that simulate dynamics from ones that match textures. The futures engine is video prediction; the social-inference sibling is visual commonsense reasoning.

The one idea you need first. Violation of expectation (VoE) shows a model two continuations — one possible, one physically impossible (a ball passing through a wall) — and checks for surprise: longer predicted looking time, higher prediction error, or explicit impossibility flags on the wrong one. Clever models fail by finding both equally plausible; physical ones flinch at the impossible.

Think of It Like This

A toddler versus a photo album

Show a toddler a tower with its middle block yanked out in a photo and she laughs — impossible, blocks need support. A photo album trained classifier captions it "tower" with full confidence. The toddler runs a physics simulator over everything she sees; the album matches patterns.

Toddler experiments are the field's gold standard for a reason: hundreds of precisely staged topplings gave developmental psychology the inventory (support by six months, occlusion permanence earlier). Machine VoE probes replay that inventory against models, asking whether silicon infancy recapitulates the carbon kind.

How It Actually Works

Two approaches compete. Mental simulation runs an approximate physics engine over parsed objects (positions, velocities, masses guessed from appearance) and reads off outcomes — strong on collisions and support, brittle on parsing errors. Learned dynamics trains video predictors on physical scenes and probes their surprise; it needs no parser but inherits every shortcut habit of its training distribution. Benchmarks (filtering towers, bouncing balls, pouring liquids) grade both with matched possible/impossible pairs so texture matching scores chance.

Hybrid systems parse-then-simulate for rigid scenes and learn dynamics for fluids and piles where engines bog down.

Worked example

Ball thrown at $10$ m/s, $30$ degrees above horizontal. Components: $v_x = 10\cos 30^\circ = 8.66$ , $v_y = 10\sin 30^\circ = 5.0$ m/s. With $g = 9.8$ :

Time to apex: $5.0 / 9.8 \approx 0.51$ s.
Apex height: $5.0^2 / (2 \times 9.8) = 25 / 19.6 \approx 1.28$ m.
Range: $2 \times 8.66 \times 0.51 \approx 8.84$ m.

A physics-simulating model predicts landing near $8.8$ m; a pattern matcher guessing from launch-frame texture might say $6.0$ m. The $2.8$ -meter gap between dynamics and guessing is what VoE probes monetize.

Code

The point of this snippet is the dynamics baseline: closed-form projectile math any learned model must rival.

1import math2
3
4def projectile(speed: float, angle_deg: float, g: float = 9.8) -> tuple:5    vx = speed * math.cos(math.radians(angle_deg))6    vy = speed * math.sin(math.radians(angle_deg))7    t_apex = vy / g  # vertical velocity spent8    height = vy**2 / (2.0 * g)9    return round(height, 2), round(2.0 * vx * t_apex, 2)10
11
12print(projectile(10.0, 30.0))  # -> (1.28, 8.84)

Watch Out For

2-D heuristics that fail on depth

Support and containment live in 3-D: a block "on" a tower in pixels may hover behind it in depth. Models reasoning over 2-D boxes call floating towers stable and flag stable ones falling, failing probes their training accuracy never hinted at.

Reason over 3-D structure (depth, point clouds, inferred volumes) for support and containment questions. Pixel support heuristics are stage makeup that depth removes.

Sim-only physics that never touched texture

Engines simulate perfect blocks on clean planes; real videos show worn toys under warm lamps. Models trained purely in simulation ace VoE probes rendered by the same engine and stumble on filmed probes where perception, not physics, is the bottleneck.

Test on real footage (or photorealistic renders) separately from simulated probes. A physics module is only as good as the parser feeding it, and sim parses are charity cases.

The Quick Version

Physical reasoning covers permanence, support, collision, and containment read from video.
Violation-of-expectation probes pit possible against impossible continuations to detect surprise.
The worked fixture lands a $10$ m/s throw at $8.84$ m with apex $1.28$ m; guessers say $6.0$ .
Mental simulation handles rigid scenes; learned dynamics cover fluids and piles.
Reason support in 3-D, and test sim-trained physics on real footage before trusting it.