AI & ML
MUSE: Multi-Tenant Model Serving With Seamless Model Updates
arXiv cs.DC0 views
arXiv:2602.11776v4 Announce Type: replace-cross Abstract: In binary classification systems, decision thresholds translate model scores into actions. Choosing suitable thresholds relies on the specific distribution of the underlying model scores but also on the specific business decisions of each client using that model. However, retraining models inevitably shifts score distributions, invalidating existing thresholds. In multi-tenant Score-as-a-Service environments, where decision boundaries reside in client-managed infrastructure, this creates a severe bottleneck: recalibration requires coordinating threshold updates across hundreds of clients, consuming excessive human hours and leading to model stagnation. We introduce MUSE, a model serving framework that enables seamless model updates by decoupling model scores from client decision boundaries. Designed for multi-tenancy, MUSE optimizes infrastructure re-use by sharing models via dynamic intent-based routing, combined with a two-level score transformation that maps model outputs to a stable, reference distribution. Deployed at scale by Feedzai, MUSE processes over 4 thousand events per second, and over 55 billion events in the last 12 months, across several dozens of tenants, while maintaining high-availability and low-latency guarantees. By reducing model lead time from weeks to minutes, MUSE promotes model resilience against shifting attacks, saving millions of dollars in fraud losses and operational costs.
Source:https://arxiv.org/abs/2602.11776PreviousA Flat KV Cache Storage Architecture for LLM ServingNextMOLT: A Hugging Face-Native, Minimal, and Scalable Framework for Agentic Reinforcement Learning
Related posts
How to detect user frustration in chatbot conversations
0dev.to
Mình Đã Biến Gemini Thành Trợ Lý Cá Nhân Như Thế Nào (Lịch, Drive, Gmail và Những Lần Suýt Toang)
0dev.to
Im looking fr any free ai models / tools are there?, leaVE UR Lights💡
0dev.to
I checked 10 SEO agency websites for AI-search readiness. The real bugs were the boring ones.
0dev.to

How much GPU memory does an LLM need? The whole sum, worked out on real cards
0dev.to

Routing Upscaling Jobs by Output Size, Not Input Size
0dev.to