AI & ML
A Feasibility Condition for Confidence-Gated Tiered LLM Inference
arXiv cs.DC0 views
arXiv:2611.05394v1 Announce Type: cross Abstract: Tiered inference distributes large language model computation across a hierarchy of devices, from on-device and edge hardware to the datacenter, and is increasingly offered as the route to cheaper, lower-latency, and more private serving. Most such systems orchestrate frozen model checkpoints, and most decide what to resolve locally and what to escalate using a confidence signal produced by a forward-deployed model. We give a closed-form condition for when no threshold on that signal makes a tiered cascade worthwhile relative to sending every query to the backstop. Modeling the routing decision as a binary test and separating local from escalation cost through an architecture cost ratio $\kappa$, we derive a feasibility condition bracketed by a signal-free floor, below which a cascade is feasible at any signal quality, and a perfect-signal ceiling, above which none is. Read along the $\kappa$ axis the condition inverts: where local compute is billed, cost binds, matching a recent empirical finding; where it is sunk, on decentralized hardware, feasibility becomes unconditional and signal quality instead sets what the cascade achieves. We turn the condition into three design requirements that existing trained-in primitives satisfy only in isolation, and we propose, as future work, a co-designed architecture and a protocol to test it. The condition and the inversion hold independently of that architecture; they are the paper's contribution.
Source:https://arxiv.org/abs/2611.05394PreviousUncertainty-Aware Resource Orchestration for Agentic AI SystemsNextUnlocking Turing Tensor Cores: A Measured Throughput Account of From-Scratch Pre-training on a Single 22G RTX 2080 Ti
Related posts
3M To Pay Over $7 Billion To Settle 'Forever Chemical' Groundwater Claims
0Slashdot
Tell HN: Google Play will public-share your address and phone if you sell in EU
0Hacker News Ask
Meta Quest Promo Codes and Coupons for November 2026
0Wired
BullMQ in NestJS: Retries, Idempotency and Dead-Letter Queues
0dev.to
80.6% at Chores, 12% at Real Work: AI's Benchmark Week
0dev.to
Efficient estimation of regression models for recurrent events in presence of terminal events
0arXiv stat.CO