Neuro-Symbolic Vision

Detectors output boxes; questions need facts. Neuro-symbolic vision parses pixels into objects plus relations, then answers by reasoning over those facts instead of re-reading pixels.

Three blocks become on and left-of facts that answer 9 of 10 queries

Why Does This Exist?

How many red blocks sit left of the blue cube. A detector lists boxes with colors; answering needs grouping, comparing, and counting over those boxes. End-to-end networks memorize common question patterns instead and break on rephrased or longer chains.

Neuro-symbolic vision splits the job: neural front ends parse the image into symbols (objects, attributes, relations from scene graph generation), and a symbolic executor runs the question as a program over those facts. New question types need new programs, not new training. The umbrella is causal and structured vision.

Think of It Like This

A librarian with a card catalog

A visitor asks for every novel set in one city and translated twice. Searching shelves by memory fails; consulting the card catalog (author, setting, editions per book) answers exactly. The catalog took work to build, but each new question is cheap.

Detectors build the catalog; the executor is the librarian. The analogy stops at catalog errors: a misfiled card misleads every later question silently, so symbol accuracy bounds the whole system and must be measured first.

How It Actually Works

The front end detects objects with attributes and pairwise relations, emitting a small fact table per image. Questions parse into programs (filter by color, relate left-of, count, compare counts). The executor runs the program deterministically over the facts. Training supervises the front end on boxes and relations; programs come from templates or a small parser, needing far less data than end-to-end answering.

Worked example

A demo image holds 3 blocks (red cube, blue cube, green sphere) with facts: on(red, table), on(blue, table), left-of(red, blue). Ten queries run over these facts: 6 counting, 4 relation checks. The system answers 9 of 10; the miss traces to one relation error (left-of flipped by 6 pixels of overlap). Front-end relation accuracy on 50 demo pairs is 0.92, and every downstream error traces to those 4 flipped pairs. Fix the parser input, not the executor.

Watch Out For

Blaming the reasoner for parser errors

A wrong answer with correct logic still fails, and teams debug the executor for weeks. The symptom is programs that read correctly while facts stay wrong. Log the fact table per query and check symbols before touching reasoning code.

The Quick Version

Neuro-symbolic vision parses images to facts, then executes questions as programs.
New question types cost new programs, not retraining.
On the demo blocks the system answers 9 of 10, missing one flipped relation.
Relation accuracy at 0.92 bounds everything downstream.
Log facts per query and debug symbols before reasoning.