AI & ML
Set-Valued Routing for Language-Model Agent Systems: A WildChat Benchmark and Cost-Aware Evaluation
arXiv cs.IR0 views
arXiv:2606.28925v3 Announce Type: replace-cross Abstract: Language-model agent systems must map natural-language requests to one or more specialized agents before planning or tool execution. This decision is often treated as top-1 selection, although compositional requests can require several capabilities and unnecessary dispatch raises cost. We formulate fixed-catalog routing as set-valued language understanding and construct a benchmark of 3,000 WildChat prompts over 12 capability-defined agents. Reproducible schema-guided reference sets are balanced by agent frequency and route cardinality. We compare profile retrieval, multilabel classifiers, a fine-tuned text encoder, and catalog-conditioned zero-shot GPT-4o. Evaluation combines set quality, latency, capability coverage, and ordinal-cost utility. The encoder attains 89.59 F1, compared with 71.79 for linear SVM and 41.51 for GPT-4o, and retains 88.02 F1 under a harder semantic-group split. We further introduce Weighted Agent Routing (WAR), a deterministic post-scoring rule that changes the quality-cost operating point without retraining; Encoder+WAR obtains the best constrained utility. The results show that catalog-specific supervision and explicit set construction are important for multi-agent dispatch, while cost-aware decision layers can complement strong language encoders.
Source:https://arxiv.org/abs/2606.28925PreviousAgentSelect: Benchmark for Narrative Query-to-Agent RecommendationNextWhere Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification
Related posts
Bikers and Bug-Lovers Join Forces to Fight a New Google Data Center
0Wired
![Localization Mapping and SLAM — Ch.21: Digital Twins [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq9x9lwbx06i8golssjdt.png)
Localization Mapping and SLAM — Ch.21: Digital Twins [KR]
0dev.to
![Localization Mapping and SLAM — Ch.24: Industrial SLAM Projects [KR]](https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fk47nuhjmx7lrx5sofla1.png)
Localization Mapping and SLAM — Ch.24: Industrial SLAM Projects [KR]
0dev.to
Android Developer Verification for Apps Outside Google Play
0dev.to
Appearance versus reality, from three lines in an interview
0dev.to
In the AI Era, the Most Valuable Engineering Skill is Taste
0dev.to