AI & ML
DiagDBench: Evaluating Agentic LLMs on Interactive Database Diagnosis and Anomaly Remediation
arXiv cs.DB0 views
arXiv:2611.07339v1 Announce Type: new Abstract: Database diagnosis and anomaly remediation are essential to reliable database management system (DBMS) operation. Despite advances in large language model (LLM)-based database agents, constructing live DBMS environments that reproduce diverse anomalies at scale remains challenging, limiting the availability of executable samples for database agent training and evaluation. To this end, we introduce DiagDBench, a large-scale benchmark for training and evaluating database agents on interactive diagnosis, anomaly remediation, and optional workload performance optimization across diverse, realistic anomaly scenarios. Each sample pairs a user request, comprising an issue description and an anomaly-verified workload under which the issue was observed, with a self-contained, executable DBMS environment that supports real-time monitoring, interactive diagnosis, remediation, and workload execution. To construct these samples at scale, we propose an LLM-based framework with two complementary pipelines: Manual-Guided Anomaly Synthesis, which sequentially synthesizes user requests, database modifications, and anomaly-injection scripts grounded in DBA manuals and official DBMS documentation; and Incident-Guided Anomaly Reproduction, which reconstructs anomaly samples end to end from real-world enterprise online diagnosis incidents. Built using this framework with sample filtering, expert review, and refinement, DiagDBench comprises 6,940 synthesized training samples and a held-out test set of 454 expert-annotated samples, spanning three hardware configurations, two DBMSs, and three database schemas. Extensive evaluations of database agents powered by multiple LLMs demonstrate that even the best-performing agent, ReAct with GPT-6 Astra, scores only 56.24, whereas most agents score below 45, underscoring the difficulty of DiagDBench.
Source:https://arxiv.org/abs/2611.07339PreviousRecursively Testing and Vibe Debugging a Google AI Studio Generated Vibe-Coded Database Software ApplicationNextSelective Retrieval Routing for Multi-Violation Knowledge Graph Repair
Related posts
California’s proposed ‘billionaire tax’ fails to pass
0TechCrunch
UK Biobank 8만 단백질 데이터, 평면 화면으론 답이 안 보였다 — VR 분자 실험실에서 길을 찾다
0dev.to
TryClothAI – Free AI Clothes Changer & Virtual Try-On
0dev.to AI

Anthropic flipped the script: Why Fable 5.2 drop changes everything for builders
0dev.to AI
Thin Cursive vs Handwriting Cursive: Which One to Use
0dev.to WebDev
BESS cybersecurity and NERC CIP: what battery storage owners should check
0dev.to Security