LLM Routers for AI Systems: Architecture, Governance, Multi-Agent Systems, Swarms, and Security

DOI: 10.5281/zenodo.23763978

From cost optimization to governed decisions about models, data, and trajectories

SGAEIA Research Series — Article 24

Aridio Silva — Independent Researcher, Brazil

Creator of SGAEIA — Secure Governed Autonomous Edge Intelligence Architecture

ORCID: 0009-0008-2411-6995

29 October 2026

© 2026 Aridio Silva. Text and original illustrations licensed under CC BY 4.0.

Abstract

Language model routing distributes requests across models, providers, and configurations with different capabilities, costs, and response times. Its usefulness, however, depends on two distinct decisions: establishing which alternatives are admissible for the task and selecting the most suitable configuration from that set. This article presents a technical review and argues that, in agentic and multi-agent systems, routing forms part of a decision boundary concerning authority, data, resources, and policy continuity. The analysis distinguishes routers, gateways, and control planes; compares rules, classifiers, similarity, cascades, and model-driven selection; and examines what benchmarks establish about quality, cost, latency, and robustness. Its analytical contribution is an SGAEIA-related framework for assessing governed routing, applied to model substitution, budgets, and multi-agent execution. The analysis explains why savings per call do not equal savings per task, availability does not establish admissible substitution, and average accuracy does not demonstrate trajectory integrity. Adoption should depend on observed gains against simple baselines, accompanied by sufficient evidence about the boundaries and consequences of decisions.

Keywords: AI; LLM routing; AI gateway; multi-agent systems; security; governance; evaluation; SGAEIA.

Contents

• Abstract
• 1. The problem extends beyond choosing the cheapest model
• 2. Routers, gateways, and control planes
• 3. How the evidence was organized
• 4. Routing strategies and their trade-offs
  • 4.1 Selecting before generation
  • 4.2 Deciding after observing a response
  • 4.3 Adapting policies and using models to select models
• 5. Quality, cost, and latency need a common unit of analysis
  • 5.1 The relevant cost is the cost of a completed task
  • 5.2 Measuring the trade-off without confusing its scales
• 6. Integration and infrastructure selection
• 7. What benchmarks demonstrate
• 8. Cross-cutting comparison of strategies
• 9. Identity, authority, and governed substitution
• 10. Agents, multi-agent systems, and swarms
• 11. Governance, security, and evidence
  • 11.1 A mediation point also concentrates risk
  • 11.2 Compliance requires context and responsibility
• 12. An SGAEIA framework for assessing governed routing
  • 12.1 The analytical contribution and its boundary
  • 12.2 Application to three engineering decisions
• 13. Evolution and maturity should be assessed by dimension
• 14. When to adopt routing and when to retain a simpler solution
• 15. Discussion and analytical limits
• 16. Conclusion
• Appendix A — Technical glossary
• References

1. The problem extends beyond choosing the cheapest model

An application using language models must decide where to invest inference capacity. A model suitable for extracting fields from a document may be insufficient to verify a complex solution; a model capable of that verification may have costs or response times unsuitable for repetitive work. Routing research explores this heterogeneity by distributing queries across alternatives rather than maintaining one choice for all traffic. The economic opportunity arises from model complementarity and task diversity, but depends on being able to anticipate or recognize where each alternative performs adequately [1][5][6].

The problem changes when the generated response participates in an action. In a question-answering assistant, an unsuitable selection may produce an unsatisfactory answer; in an agent that queries systems, writes code, or delegates work, the same decision may change the data transmitted, tools used, and subsequent effects. Model selection must therefore account for task state, the person or service authorizing execution, and the boundaries accompanying that authorization. SGAEIA provides the architectural perspective adopted here: the capability to produce a response and the authority to carry out an action are different properties [27].

The central thesis is that routing in agentic systems extends beyond inference optimization and forms part of a critical boundary for decisions, governance, and evidence production. This does not turn every router into a security system or make a learned algorithm mandatory. Shafran and colleagues already place routers within LLM control planes and demonstrate attacks targeting model selection [9]. The step developed here is to organize the implications of that understanding into comparable criteria, engineering decisions, and verifiable evidence.

Six questions guide the analysis: when routing saves money while preserving quality; when a strategy outperforms simple alternatives; what value a gateway provides independently of its algorithm; what risks follow from concentrating credentials and policies; what changes with agents and swarms; and what evidence supports governed selection, substitution, and revocation. The article contributes a synthesis of these questions within a common decision framework, presented and applied in Section 12. Its value is analytical: examining a configuration, identifying its assumptions, and explaining a decision to adopt or reject it. The effectiveness of a particular implementation requires evaluation in its intended environment.

2. Routers, gateways, and control planes

An LLM router is a selection function: it receives information about a task and chooses a model, deployment, or execution configuration. The function may use local rules, a classifier, a similarity calculation, or a policy informed by intermediate results. It does not require an HTTP server or public API; selection can occur within the application process. The sequential-decision formulation used by LLMRouter helps explain cases involving history, personalization, and multiple steps [8].

An AI gateway mediates access to models. It may adapt protocols, check credentials, enforce limits, forward requests, and produce telemetry, while also hosting a routing function. These responsibilities remain useful even if every query goes to the same model. LiteLLM's virtual-key documentation, for example, describes control of model access and spend tracking; these are mediation capabilities, not evidence that its selector achieves the best cost-quality trade-off [10].

The control plane administers decisions governing the environment: policies, configurations, permissions, versions, and substitution criteria. The data plane carries the corresponding traffic. A gateway helps enforce these decisions, but its administrative components, inference path, and evidence storage need not share the same privilege domain. The distinction has practical consequences: compromising a client application should not permit editing gateway policies, and observing a response should not grant access to the credentials enabling it [16].

Table 1. 2. Routers, gateways, and control planes.

Figure 1. The flow distinguishes task analysis, routing decisions, and result validation. Selection should consider applicable constraints before optimizing performance. This is a conceptual diagram: the presence of these blocks does not establish correct policy enforcement in a deployment.

Conflating these layers creates two evaluation errors. The first attributes infrastructure benefits, such as key inventories or centralized auditing, to the selection algorithm. The second assumes that a feature-rich gateway has already solved agent quality, authorization, and security. A coherent architecture defines each component's responsibility and verifies their interactions, including when a component is unavailable or operates with incomplete information.

3. How the evidence was organized

The analysis is a narrative technical review guided by the six introductory questions. The starting corpus was the Article 24 material, supplemented by original studies, publication records, official documentation, risk-management references, and technical incident reports. Selection prioritizes sources that explain a mechanism, provide comparable evaluation, or document a relevant operating condition. The reference date is 29 October 2026; it does not represent an exhaustive search of all routing research.

Four types of statement are distinguished. Method and benchmark results are attributed to their authors and evaluated conditions. Product features are described as capabilities documented by suppliers, while legal and risk-management requirements remain linked to their official sources. Cross-cutting comparisons, the governance matrix, and decision scenarios are analytical syntheses developed in this article. This separation distinguishes third-party observations from reasoned implications and proposed assessment approaches.

An identical savings percentage can represent fewer calls to an expensive model, lower API charges, or reduced computational cost. Comparisons therefore retain the unit, baseline, and generalization boundary rather than ranking studies by their largest percentage. They also distinguish studies proposing a method from work re-evaluating several methods under a common protocol. Neither category removes the need to examine data, model pools, evaluators, and potential commercial interests.

The article does not present published results as its own experiments. The calculations in Section 5 are hypothetical scenarios with explicit inputs and reproducible arithmetic. Its contribution is demonstrated through a consistent comparison and application of the governance framework to decisions for which isolated criteria yield insufficient conclusions. Empirical assessment of risk reduction, control performance, or production security belongs to a subsequent stage with its own protocol and results.

4. Routing strategies and their trade-offs

4.1 Selecting before generation

Predictive routing chooses a destination using information available before the response: query content, domain, length, identity, history, or a vector representation. Explicit rules can send documents of a particular class to an approved deployment or reserve a model for tasks requiring a specific capability. Their transparency helps explain decisions and examine exceptions; their limitation is representing complex boundaries without proliferating rules. A keyword rule, for example, may confuse a programming question with a request that merely mentions a programming language.

Classifiers and preference models replace some rules with a learned estimate. RouteLLM uses preference data to estimate when the more capable model is worthwhile and investigates different selection architectures [1]. Hybrid LLM likewise adjusts routing between smaller and larger alternatives according to a desired quality level [4]. The advantage is selecting without generating all candidate responses; the dependency is representative supervision and recalibration when models, tasks, or quality criteria change.

Similarity routing compares the current task with previous examples and uses their history to recommend a model. This exploits domain structure, but proximity in embedding space does not establish equal difficulty or consequences. A simple question and a critical question can share vocabulary while requiring different treatment. Similarity may inform expected performance; authorization to process the data must come from separate criteria.

4.2 Deciding after observing a response

A cascade consults a model before deciding whether to stop or escalate. FrugalGPT explores combinations of models and acceptance criteria to reduce inference cost [2]. The strategy is attractive when the first alternative solves a substantial share of tasks and its result can be assessed reliably enough. When almost every task escalates, the system pays for the initial model and its successor, plus evaluation and accumulated latency.

AutoMix combines an initial answer, self-verification, and decision-making under uncertainty, recognizing that intermediate assessment can be noisy [3]. The central issue extends beyond the smaller model's capability to whether the verifier recognizes its errors. Valid JSON may contain incorrect data, and a confident response may lack support. Format validation, content assessment, and action authorization are different checks, even when they share infrastructure.

An LLM judge offers flexible evaluation but adds cost, dependency, and potentially correlated errors. Changing the judge's supplier may increase diversity without establishing effective independence: instructions, references, and criteria may remain shared. For important tasks, the system must measure disagreements, false acceptances, and false rejections alongside average scores. Evaluator identity should be part of the decision record, especially when its score determines a subsequent action.

4.3 Adapting policies and using models to select models

Adaptive strategies may update decisions from observed results, while generative strategies use an LLM to interpret, decompose, or plan selection itself. Bandits and related approaches face an additional trade-off: exploring alternatives to learn may incur costs and inferior responses during exploration. A generative selector adds inference before the main inference and can make decisions harder to explain. These methods must justify their additional costs and risks against a simple alternative.

Jiangeng and colleagues organize the field into predictive, adaptive, and generative operations and distinguish infrastructure work that evaluates or supports routing [7]. This complements the distinction between decisions made before and after generation. A gateway may combine mandatory rules, a predictive classifier, and adaptive escalation without those mechanisms competing in every respect. A taxonomy should identify the decision each mechanism makes and the information it uses.

Figure 2. The taxonomy relates advance selection, cascades, and intermediate evaluation. It helps locate additional calls and supervision dependencies. Categories can coexist within an application; their presence establishes no universal ranking of quality.

Table 2. 4.3 Adapting policies and using models to select models.

5. Quality, cost, and latency need a common unit of analysis

5.1 The relevant cost is the cost of a completed task

The price of a call does not describe system cost. The total includes input and output tokens, retries, embeddings, verifiers, infrastructure, and operational work. In agentic workflows, it also depends on the number of calls needed to complete a task and how many trajectories must be repeated. A model with cheap tokens may make a task more expensive if it requires more corrections, generates unnecessarily long responses, or increases tool calls.

A useful decomposition, proposed here for operational assessment, is:

This expression organizes the components to measure; it does not assume they are constant or independent. Retry cost depends on answer quality and recovery policy, while operating cost may contain a significant fixed component. Alternatives must be compared using the same definition of task completion and the same acceptance criteria. Otherwise, apparent savings may simply reflect accepting inferior responses or stopping tasks prematurely.

Consider a hypothetical scenario in which a compact-model call costs US$0.002 and a more capable-model call costs US$0.020, with input and output lengths already incorporated into these prices. A selector sends 80% of queries to the former and 20% to the latter, adding US$0.0004 per query for selection. Average variable cost is US$0.006, 70% below sending all queries to the more expensive model. The calculation assumes unchanged traffic distribution, per-call costs, and required quality.

If operating the router adds US$500 monthly, the variable reduction does not translate fully into total savings. At 10,000 queries, spending rises from US$200 to US$560; at 50,000, it falls from US$1,000 to US$800. Break-even is approximately 35,715 monthly queries under these assumptions. The example demonstrates why volume and fixed cost must accompany a savings percentage presented to decision-makers.

Table 3. 5.1 The relevant cost is the cost of a completed task.

This break-even expression applies only when routed variable cost is lower than the baseline and the fixed component can be represented by F. It changes if the gateway serves other objectives, such as access management, whose costs would exist without dynamic selection. In that case, the incremental cost of enabling routing should be assessed, avoiding double allocation of infrastructure costs. The economic decision becomes more precise when mediation value and selector value are separated.

5.2 Measuring the trade-off without confusing its scales

RouteLLM defines PGR as the fraction of the performance gap between two models recovered by the router. APGR summarizes the recovery curve as expensive-model use varies, while CPT identifies the call fraction needed to achieve a specified recovery [1]. These measures help assess a binary configuration but do not automatically equal lower dollar charges. Actual cost also depends on generated lengths, tariffs, and auxiliary operations.

The expression requires a meaningful, nonzero denominator. PGR of 50% means recovering half the gap between references, not achieving 50% of the stronger model's score. If reference scores are close, small evaluation changes may substantially alter the indicator. Absolute scores, variability, and the cost curve should therefore accompany the normalized value.

RouterBench and LLMRouterBench extend the analysis to model pools and cost-quality frontiers [5][6]. A configuration is dominated when another provides at least equal quality at no greater cost, improving at least one dimension. The Pareto frontier contains non-dominated alternatives, but its position depends on the compared pool and chosen measures. Being on that frontier does not establish compliance with authorization, data-residency, or security requirements.

Latency must include selection, queues, networking, generation, and validation. Cascades may accumulate waiting time; parallel execution may shorten time at the cost of simultaneous calls. Time to first token and total response time should both be observed, with percentiles such as p50, p95, and p99 under a stated load. Parameter count alone does not determine perceived speed: infrastructure, concurrency, context, and output length can change the comparison.

6. Integration and infrastructure selection

A common API reduces integration work, but syntactic compatibility does not establish functional equivalence. Models and providers may differ in tool handling, structured outputs, images, context, sampling parameters, and reasoning configurations. Applications must verify the capabilities they use and define how missing features are handled. A successful transport-level substitution may fail at task semantics even when the HTTP response indicates success.

Figure 3. The reference architecture organizes mediation and observation responsibilities around selection. Components represent conceptual roles that may be distributed across services and privilege domains. A gateway does not replace tool controls or authorization of actions performed by the application.

Reasoning effort can be part of the selected configuration when the provider exposes that control. The evaluated unit then includes model, version, configuration, and execution conditions rather than just a model name. The same family may produce different costs and times depending on effort, context, and response length. Accepted values and their meaning must come from documentation for the version used, rather than being assumed from a multi-provider interface.

The ecosystem also contains tools with different purposes. RouteLLM and LLMRouter support development and evaluation of selectors [1][8]; LiteLLM documents access mediation and key management [10]; OpenRouter documents model fallback and provider selection [12][13]. Portkey describes gateway features including resilience and traffic-control options [15]. Documentation establishes the advertised capability and its stated conditions, but does not independently establish suitability for the reader's environment.

Table 4. 6. Integration and infrastructure selection.

Self-hosting does not establish sovereignty if embeddings, telemetry, or support mechanisms transmit data to third parties. A managed service should likewise not be assessed solely by model count: actual location, retention, subprocessors, administration, and evidence export matter. Candidate lists should identify deployments rather than only commercial model names. Two deployments of the same model may have different data-processing conditions.

Gateway performance must be measured separately from model time. Tests against a mock server help isolate mediation overhead but do not describe the complete application; authentication, databases, logs, filters, and concurrency change the result. LiteLLM's benchmark documentation should be read with its configuration and load conditions [14]. Supplier comparisons additionally require scrutiny of versions, configuration parity, and commercial interests before concluding that millisecond differences will persist in production.

7. What benchmarks demonstrate

The reviewed studies support the possibility of saving through model selection, but their findings answer different questions. FrugalGPT reports cost reductions of up to 98% in cascade configurations studied by its authors [2]. Hybrid LLM reports up to 40% fewer calls to the larger model, while AutoMix reports over 50% lower computational cost at comparable performance in its evaluated settings [3][4]. These numbers do not rank the methods: they measure different objects and use different tasks, models, and evaluators.

RouteLLM demonstrates the potential of learning selection from preferences. Its matrix-factorization configuration augmented with judge data achieves MT-Bench CPT of 13.40% to recover 50% of the performance gap and 31.31% to recover 80% [1]. Figure 4 represents this comparison between call fractions and gap recovery. The finding demonstrates efficiency under that protocol; it does not determine the appropriate fraction for another domain or generation of models.

Figure 4. The comparison uses MT-Bench results from RouteLLM. CPT concerns recovery of the gap between two references rather than a percentage of the stronger model's absolute score. Values belong to the study's model pair, data, and evaluation [1].

RouterBench contributes precomputed inference outcomes and a protocol for comparing policies in cost-quality space [5]. This design lowers the cost of testing selectors on the same responses and identifies gains without requiring new generation for every evaluation. A static matrix, however, does not automatically reproduce queues, outages, API changes, or adaptations within a real trajectory. The benchmark makes a comparison controllable; operational assessment must add missing dimensions.

LLMRouterBench offers an important counterpoint to method-proposing studies. Its protocol covers 33 models and 21 datasets, confirms model complementarity, and finds several methods with similar performance, including cases that do not reliably beat simple baselines [6]. The useful conclusion is that complementary alternatives do not ensure a selector can exploit them. Improving the model pool and data coverage may matter more than changing the algorithm's name.

Table 5. 7. What benchmarks demonstrate.

Local evaluation should include random routing, always the lowest-cost model, always the best fixed model for the sample, simple rules, and a small classifier. A retrospective oracle can estimate pool potential using response information unavailable before execution. Such an oracle is an analytical reference, not a cost-free deployable system. Comparisons must use the same data, models, and acceptance criteria, separating calibration data from the data used to decide whether the strategy delivered an improvement.

8. Cross-cutting comparison of strategies

Strategies should not be selected solely by their best published score. The same method may suit a repetitive task and be unsuitable where verification is difficult or error consequences are substantial. The following framework makes that difference explicit by comparing operational needs, validation behavior, and governability. Classifications are an engineering synthesis based on the mechanisms discussed, not results of a common experiment conducted in this article.

Table 6. 8. Cross-cutting comparison of strategies.

Table 7. 8. Cross-cutting comparison of strategies.

The comparison explains why auditability and quality are related but distinct. A rule can be easy to explain yet select poorly; a classifier can choose well within one distribution but be difficult to calibrate elsewhere. It also explains why useful complexity has no fixed order: maintaining hundreds of exceptions may cost more than operating a stable classifier. Selection should consider the full configuration, operation, investigation, and update lifecycle.

For agents, the decisive question is whether a strategy considers information relevant to the next action. Swarms additionally involve concurrency, shared state, and accumulated consequences. Access policies and collective limits must remain applicable regardless of the selection method. This separation allows algorithm improvements without making every update an implicit expansion of authority.

9. Identity, authority, and governed substitution

A valid key identifies a credential accepted by the gateway. By itself, it does not establish whether its holder may transmit particular data, use a deployment, or initiate an action. The distinction between authentication and authorization is central to Zero Trust architecture [16]. Agentic systems must also identify on whose behalf an agent acts and which portion of authority it may delegate.

Figure 5. Separating external and internal credentials reduces the need to distribute provider keys to clients. Effectiveness depends on gateway isolation, object-level authorization, and protection of administrative operations. A virtual key remains a secret with its own scope and lifecycle.

A deployment should limit destinations accessible to each user, service, or agent while protecting credentials granting that access. Client-supplied parameters must not permit replacing administrative endpoints or redirecting secrets to arbitrary hosts. Authorization must also cover log access, budget changes, and credential issuance. Concentrating secrets in a gateway reduces their dispersion but increases the potential impact of mediator compromise.

Fallback deserves particular attention because it is often presented as an availability feature. OpenRouter documents successive model attempts according to priority and error conditions, including moderation-related cases [12]. For an organization, a technically accessible alternative is not necessarily admissible for the request. A safety refusal should likewise not automatically be treated as an outage to bypass through a less restrictive model.

The admissible set can be represented as A(x, h, i, p), where x is the task, h relevant history, i the requester's identity and authority, and p current policies. Each member should represent a deployment and configuration permitted under current conditions. Quality, cost, and latency optimization takes place within that set, rather than compensating for mandatory violations with economic advantages. This high-level formalization is proposed to organize the analysis without specifying private SGAEIA mechanisms.

The weights express preferences among already admissible alternatives; they do not price missing authorization. If the set is empty, the application needs a defined response, such as suspension, reduction of permitted functionality, or referral to an authorized decision-maker. If policy, identity, or state changes, a previously valid choice may cease to be valid. Revocation must reach queues, caches, and ongoing execution according to the application's specified boundaries.

10. Agents, multi-agent systems, and swarms

An agent may interact repeatedly with a model, use tools, and revise its plan. A multi-agent system distributes some work among agents exchanging information or sharing resources; a swarm may intensify decentralization and collective coordination. Agent count alone does not determine criticality. One broadly privileged action may be riskier than many limited queries, while the combination of apparently small actions may produce an effect not visible in any individual action.

Figure 6. Role distribution illustrates how agents can need different capabilities. Planning, execution, and review do not by themselves establish effective separation of duties. Assessment must also consider authority, dependencies, memory, collective budgets, and execution containment.

Selecting by role is a first approximation: triage may need low latency; planning, greater capacity to assess alternatives; execution, format adherence; review, verification capability. These associations must be evaluated by task, avoiding assumptions that the most expensive model is always the best reviewer or a small model sufficient for all triage. An early misclassification may route data or authority incorrectly before any review occurs. History-aware routing provides more information but does not replace access controls [8].

In long trajectories, measuring i...