Core ML

Edge Deployment

Instead of sending your data to a massive server in the cloud, edge deployment shrinks the model so it can run directly on your phone, laptop, or IoT device.

Why Does This Exist?

When you build a machine learning feature, the default architecture is to host a massive model in a cloud data center and have your app talk to it via an API.

But cloud inference breaks down in three specific scenarios:

Latency. If you are building a self-driving car or a live translation earpiece, you cannot wait 200 milliseconds for a round-trip to us-east-1.
Privacy. If you are building a keyboard app or a medical symptom checker, sending raw, unencrypted keystrokes to a cloud server is a massive security liability.
Cost. If you have millions of daily active users, paying a cloud provider for every single token they generate will bankrupt you.

Edge deployment fixes all three. By moving the model directly onto the user's hardware—their phone, their laptop, or an IoT sensor—you eliminate the network hop, keep the data completely private, and push the compute cost onto the user's battery rather than your AWS bill.

Think of It Like This

Imagine a restaurant where every time a customer asks a waiter a question, the waiter has to call the head chef in Paris on the phone to get the answer. The answers are brilliant, but the phone calls are expensive, slow, and if the internet goes down, the restaurant grinds to a halt.

Edge deployment is like training the waiter to memorize a condensed version of the menu. They might not know the exact farm the tomatoes came from, but they can answer 95% of questions instantly, for free, without a phone line.

How It Actually Works

You cannot simply copy a 70-billion parameter LLM onto an iPhone. An iPhone has limited RAM and a battery that drains. Edge deployment is entirely an exercise in brutal compression.

1. Shrinking the Model

Before a model can go to the edge, it must be aggressively compressed.

Quantization: Converting the 32-bit floating-point weights into 8-bit or 4-bit integers (model-quantization). This slashes the memory footprint by 4x to 8x.
Pruning: Deleting connections in the neural network that contribute the least to the output (model-pruning).
Distillation: Training a tiny model to mimic the outputs of a massive model (knowledge-distillation).

2. The Edge Hardware

Phones and laptops now ship with specialized silicon designed explicitly to run these compressed models. Apple has the Neural Engine, Qualcomm has the Hexagon NPU, and Google has the Edge TPU. These chips run matrix math with incredible efficiency, burning far less battery than running the same model on the main CPU.

3. The Formats

You don't deploy PyTorch files to the edge. You convert the model into an optimized format targeting the specific hardware. ONNX Runtime, Apple's CoreML, and Google's TensorFlow Lite are the standard bridges between your training code and the user's silicon.

Watch Out For

The update cycle nightmare. When you host a model in the cloud, you can deploy a new version at 2:00 AM, and instantly 100% of your users are on the new model.

When you deploy a model to the edge, it is embedded in an app binary. To update the model, you have to ship an app update. This means you will permanently have users running versions of your model from three years ago. You cannot deprecate old input formats, and if you discover a catastrophic hallucination bug, you cannot force a fix until the user decides to visit the App Store.

The Quick Version

The Problem: Cloud inference is slow, expensive at scale, and requires sending private user data over the internet.
The Fix: Shrink the model using quantization and distillation, then run it directly on the user's hardware (the edge).
The Engine: Hardware NPUs (Neural Processing Units) and optimized formats like ONNX and CoreML make this fast and battery-efficient.
The Cost: The model is smaller and dumber, and updating it requires an app store release.