Microcontroller Vision TinyML
TinyML runs person or keyword vision on milliwatt chips, so sensors answer one question locally without sending images.
Why Does This Exist?
A door sensor that streams 96 by 96 frames to cloud burns bandwidth forever to answer one bit: person or not. Privacy breaks too, since images leave the room. Microcontroller vision exists to answer on the chip: an ESP32-S3 at 240 MHz with 512 KB SRAM and 8 MB PSRAM runs a 250 KB INT8 person model and reports only the boolean.
The stack is TFLite Micro with ESP-NN kernels plus esp32-camera. ESP-IDF 5.x carries the esp-tflite-micro component. Models are depthwise separable CNNs at 96 by 96 grayscale, fully INT8 quantized. Edge deployment places these sensors under gateways. INT8 networks explain the quantization. Inference optimization sets the budget habit first.
Basics first: memory is static. The tensor arena is a fixed buffer at compile time, no heap at runtime. Camera frame, model, arena, WiFi stack, and app logic all share 512 KB internal SRAM unless PSRAM is enabled.
Think of It Like This
A door peephole with one trained answer
A peephole does not show the hallway to the building manager. It lets the person inside answer one question: is someone there. No recording, no upload, just the answer when it changes.
TinyML sensors are peepholes. The model looks through 9 KB frames and reports person true or false. The analogy stops at learning: peepholes never improve, while these sensors still need field data collection to fix blind spots.
How It Actually Works
Size the arena first. Person detect needs about 225 KB arena for 96 by 96 INT8 plus 250 KB model flash. Sum is 475 KB against 512 KB SRAM, leaving 37 KB for stack plus WiFi, which is too tight. Enable octal PSRAM and place the arena plus frame buffers there with MALLOC_CAP_SPIRAM, keeping hot scratch internal. Boards for AI work pair S3 with 8 MB PSRAM for this reason.
Use accelerated kernels or lose 40 times. Generic TFLite Micro person detect on S3 measures about 2300 ms per frame. Espressif ESP-NN hand tuned SIMD cuts the same model to about 54 ms at 240 MHz, since vector MACs replace scalar loops. Check inference_ms in logs: near 54 means the vector path runs, near 2000 means generic kernels shipped.
Send answers, not images. A 96 by 96 grayscale frame is 9 KB since 96 times 96 equals 9216 bytes. Five per second is 45 KB per second forever. A boolean plus latency is a few bytes on change only. Report person_detected plus inference_ms over MQTT and keep images on device.
Worked sensor math
Generic 2300 ms is 0.43 fps since 1000 divided by 2300 equals 0.43. Accelerated 54 ms is 18.52 fps since 1000 divided by 54 equals 18.52, a 42.59 times gain since 2300 divided by 54 equals 42.59. SRAM leaves 37 KB since 512 minus 475 equals 37, which forces PSRAM. Frame is 9 KB since 96 times 96 equals 9216 bytes.
Code
The point is the fit check before flashing ten sensors. It recomputes fps, gain, and SRAM headroom.
import numpy as np
def fps(latency_ms: float) -> float: return 1000.0 / latency_ms # sensor answers per second
def sram_left(sram_kb: float, model_kb: float, arena_kb: float) -> float: return sram_kb - model_kb - arena_kb # headroom for stack plus radio
print(fps(2300.0)) # -> 0.43478260869565216print(fps(54.0)) # -> 18.51851851851852print(2300.0 / 54.0) # -> 42.592592592592595print(sram_left(512.0, 250.0, 225.0)) # -> 37.0Ship only with PSRAM enabled and ESP-NN confirmed. A 37 KB margin without PSRAM crashes under WiFi load.
Watch Out For
No PSRAM boards that crash like bad models
A sensor without PSRAM boots, runs one inference, then faults when WiFi plus camera plus arena collide. The symptom looks like model instability across random boards. Fix it by specifying S3 boards with 8 MB PSRAM and by asserting arena placement in PSRAM at startup.
Generic kernels that hide the 40 times gap
A clean build with upstream TFLite Micro instead of esp-tflite-micro runs 2300 ms while demos promise 54 ms. The symptom is correct answers too slow to use. Fix it by depending on the Espressif component, by enabling ESP-NN, and by logging inference_ms per release.
The Quick Version
- TinyML answers one vision question on milliwatt chips and sends booleans, not images.
- Budget 250 KB model plus 225 KB arena against 512 KB SRAM, then move to PSRAM.
- ESP-NN cut 2300 ms to 54 ms, 0.43 to 18.5 fps, 42.6 times faster on S3.
- A 96 frame is 9 KB, so local answers remove continuous uploads.
- Require PSRAM boards and log inference ms, or memory and generic kernels will decide results.