AI & ML
RecToolBench: Benchmarking Recommendation-Specific Tool Orchestration under Fuzzy User Intent
arXiv cs.IR0 views
arXiv:2609.30717v2 Announce Type: replace Abstract: Recent advances in agentic recommender systems are shifting recommender systems from passive filtering engines to instruction-following agents that use external tools to resolve user intent. However, existing benchmarks often assume explicit user intent, simplified tool environments, or isolated function calls, leaving realistic tool orchestration for recommendation underexplored. To bridge this gap, we propose RecToolBench, a Model Context Protocol (MCP)-based benchmark for evaluating tool-using recommender agents under fuzzy user instructions. RecToolBench contains more than 1,200 executable tasks across three recommendation domains, 13 MCP servers, and 32 tools, spanning single-tool calls, parallel tool calls, sequential tool chains, and hybrid tool orchestration. We construct RecToolBench with a scalable synthesize--fuzzify--judge pipeline that generates executable fuzzy recommendation tasks, and evaluates agent trajectories using rule-based execution checks and rubric-based LLM evaluation. Experiments on representative LLMs show that syntactically valid tool calls do not guarantee successful recommendations. Models struggle with semantic parameter grounding, multi-step evidence integration, and grounded final recommendations, especially as orchestration complexity increases. Our results identify tool orchestration under fuzzy user intent as a major bottleneck for agentic recommender systems. Our data and code are available at https://github.com/ShawnChenn/RecToolBench.
Source:https://arxiv.org/abs/2609.30717PreviousMERGED: Multimodal Entity Resolution via Generated Expert Reasoning DistillationNextRetrieval-Augmented Generation for Predicting Cellular Responses to Gene Perturbation
Related posts

Canva Brings Free All-in-One Affinity App to iPad
0MacRumors
GPT-Rosalind-6 System Card Addendum
0Hacker News New
Mediroute: An open-source AI platform exposing global drug price disparities
0Hacker News New
An intuition for how neural nets can understand text
0Hacker News New
An open LLM read 291 retracted papers: 7.7% of passages mention the retraction
0Hacker News New
Show HN: Maat – AI agents that can diagnose Ubuntu but can't change it
0Hacker News Show