Backend
Achieving Near-Zero-Overhead Multi-Model Hierarchical Classification in Real-Time Detection Pipelines
arXiv cs.DC0 views
arXiv:2608.11770v2 Announce Type: replace-cross Abstract: Edge-deployed vision systems in target recognition, surveillance, autonomous vehicles, and drone domains require hierarchical inference pipelines where a detection model identifies objects of interest and downstream classifiers provide fine-grained attribute analysis. Running all models on the GPU creates a serial bottleneck that limits real-time throughput as pipeline stages grow. Modern edge SoCs pair GPUs with dedicated neural accelerators (NPUs, DLAs) capable of concurrent execution, yet deploying custom models on these accelerators remains impractical due to strict operator constraints, quantization incompatibilities, and an undocumented end-to-end pipeline. We target NVIDIA Jetson DLA cores as the representative platform. We present a five-step methodology for zero GPU fallback DLA INT8 deployment of classification backbones, comprising architecture adaptation, manual dynamic range workaround to rescue TensorRT's implicit quantization (recovering 94.0% accuracy from implicit quantization's 75%) for rapid pipeline validation before explicit quantization, quantization-aware training, ONNX graph surgery for DLA compilation, and a concurrent GPU-detection/DLA-classification inference pipeline. We document nine engineering constraints with root-cause analysis and generalizable solutions. Validation on a dual-head person attribute classifier running on DLA alongside a GPU object detector on a Jetson Orin NX demonstrates near-zero pipeline overhead (12.5 vs. 13.3~FPS detector-only at 1080p), with dual-DLA scaling at no additional cost. The methodology is backbone-agnostic and generalizes to any detection-classification edge pipeline.
Source:https://arxiv.org/abs/2608.11770PreviousProperties of Concurrent Algorithms that Allow AsynchronyNextCharacterizing High Bandwidth Flash for LLM Serving
Related posts
Secure Password Storage: Why Applications Should Never Store Plain-Text Passwords
0dev.to OpenSource
Perplexity Decider v1.3 Open Weights Beat Jev 1.14
0dev.to OpenSource
I Measured My MCP Server's Context Tax Before Adding a Third Tool
0dev.to OpenSource

Let your coding agent test the API it just wrote
0dev.to OpenSource

Inside Stripe’s AI Coding Interview: Rule Engines, AI Prompts, and Tests
0dev.to Programming
From Idea to Wireframe to Production Design: An Open-Source Product Blueprint Guide
0dev.to Programming