Deploying vision models in production on edge devices (IP cameras, drones, smartphones, robots, kiosks) is a distinct discipline from cloud deployment. Latency, consumption, bandwidth, hardware cost impose specific constraints.
TL;DR
- Edge = inference on device, not cloud.
- Hardware: Jetson Orin, Raspberry Pi 5, Coral TPU, mobile.
- Optimizations: int8 quantization, pruning, distillation.
- Frameworks: TensorRT, TFLite, ONNX Runtime, OpenVINO.
Why edge vs cloud
- Latency: 10-50 ms edge vs 200-500 ms cloud (network round-trip)
- Bandwidth: avoid sending 4K video continuously streaming to cloud
- Privacy: image / video data stays on device (GDPR, privacy)
- Availability: works offline, without connectivity
- Cost: no cloud GPU bill per inference
Popular edge hardware 2026
- NVIDIA Jetson Orin Nano (~$250): 40 TOPS INT8, 8 GB RAM, ideal robotics / drones
- NVIDIA Jetson AGX Orin (~$2000): 275 TOPS, advanced vision, vehicles
- Raspberry Pi 5 (~$80): Cortex-A76, OK for light models (YOLO nano)
- Google Coral TPU (~$150): 4 TOPS dedicated ML, USB or PCIe
- Apple Silicon (Mac, iPhone): integrated Neural Engine 15-35 TOPS
- Qualcomm Snapdragon: Hexagon DSP for Android smartphones
- Intel CPU + OpenVINO: for kiosks and x86 embedded
- Hailo-8: 26 TOPS embedded, surveillance
Optimization techniques
Quantization
Convert FP32 (32-bit float) to INT8 (8-bit int):
- 4x less memory
- 2-4x faster per hardware
- Typical precision loss: 1-3%
- Post-Training Quantization (PTQ): fast, no re-training
- Quantization-Aware Training (QAT): better quality, requires re-training
Pruning
Remove weights close to zero:
- Structured pruning (entire filters): hardware speedup
- Unstructured pruning: less practical speedup
- Magnitude pruning, lottery ticket hypothesis
Knowledge distillation
Need a professional website?
Kolonell builds websites that attract clients, optimized for the Sénégalese market. Free quote in 2 minutes.
Train small model (student) to mimic large model (teacher):
- Teacher: YOLOv11-xl
- Student: YOLOv11-nano
- Student learns teacher "soft labels"
- Better performance than student trained from scratch
Deployment frameworks
- TensorRT (NVIDIA Jetson) — aggressive optimization, INT8 + FP16, performance leader
- TFLite (Google, mobile) — Android, iOS, Coral TPU
- ONNX Runtime — multi-platform, universal intermediate format
- CoreML (Apple) — iOS, macOS, Apple Silicon optimized
- OpenVINO (Intel) — Intel CPU, kiosks, NUCs
- Pytorch Mobile — Pytorch direct on device
Typical deployment pipeline
- Train model (Python, PyTorch / TF)
- Export ONNX
- Optimize (quantization, pruning)
- Convert to edge format (TensorRT, TFLite, etc.)
- Benchmark on target device (latency, memory, precision)
- Iterate if metrics insufficient
FAQ
Q: Which edge hardware to start?
A: Jetson Orin Nano (~$250) covers most cases. Coral TPU if very budget-constrained.
Q: Expected INT8 quantization speedup?
A: 2-4x typical. Per hardware: Coral TPU 4-10x (INT8-optimized TPU), CPU 1.5-2x.
Conclusion
2026 computer vision edge reached a maturity level where you can deploy YOLO, CLIP, SAM on device with acceptable latency and quality. Jetson Orin Nano, Coral TPU, and mobile ecosystem (CoreML, TFLite) cover 90% of production needs. Quantization + distillation are indispensable optimizations. For any vision team, planning edge deployment from model design is essential.
Mohamed Bah
Fondateur, Kolonell
Passionate about digital and entrepreneurship in Africa, Mohamed has been helping Sénégalese businesses with their digital transformation since 2020. Founder of Kolonell, he believes every SME deserves a professional and accessible online présence.
