Backend
Achieving Near-Zero-Overhead Multi-Model Hierarchical Classification in Real-Time Detection Pipelines
arXiv cs.DC0 views
arXiv:2608.11770v2 Announce Type: replace-cross Abstract: Edge-deployed vision systems in target recognition, surveillance, autonomous vehicles, and drone domains require hierarchical inference pipelines where a detection model identifies objects of interest and downstream classifiers provide fine-grained attribute analysis. Running all models on the GPU creates a serial bottleneck that limits real-time throughput as pipeline stages grow. Modern edge SoCs pair GPUs with dedicated neural accelerators (NPUs, DLAs) capable of concurrent execution, yet deploying custom models on these accelerators remains impractical due to strict operator constraints, quantization incompatibilities, and an undocumented end-to-end pipeline. We target NVIDIA Jetson DLA cores as the representative platform. We present a five-step methodology for zero GPU fallback DLA INT8 deployment of classification backbones, comprising architecture adaptation, manual dynamic range workaround to rescue TensorRT's implicit quantization (recovering 94.0% accuracy from implicit quantization's 75%) for rapid pipeline validation before explicit quantization, quantization-aware training, ONNX graph surgery for DLA compilation, and a concurrent GPU-detection/DLA-classification inference pipeline. We document nine engineering constraints with root-cause analysis and generalizable solutions. Validation on a dual-head person attribute classifier running on DLA alongside a GPU object detector on a Jetson Orin NX demonstrates near-zero pipeline overhead (12.5 vs. 13.3~FPS detector-only at 1080p), with dual-DLA scaling at no additional cost. The methodology is backbone-agnostic and generalizes to any detection-classification edge pipeline.
Source:https://arxiv.org/abs/2608.11770PreviousProperties of Concurrent Algorithms that Allow AsynchronyNextCharacterizing High Bandwidth Flash for LLM Serving
Related posts

PostgreSQL 19 removes the MultiXact members wall. Reclamation still waits for template0.
0dev.to
Hugging Face Breach: Security Lessons for AI Agents
0dev.to
Incus Project and Migration Trust Boundaries Need Separate Review
0dev.to

Next.js `revalidatePath` vs `revalidateTag` in 2026: Choosing the Right Cache Invalidation Primitive at Scale
0dev.to

'python' Is Not Recognized as an Internal or External Command: Every Fix
0dev.to

Xcode Instruments in Practice: Profiling and Optimizing iOS Apps
0dev.to