AI & ML
PS-HalluBench: Taxonomy-Driven Benchmarking of LLM Hallucinations in Power Systems
arXiv cs.SY0 views
arXiv:2611.22012v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly explored for power-system applications, yet their reliability in engineering analysis remains uncertain. We introduce PS-HalluBench, a manually constructed benchmark of 200 questions that evaluates hallucination-related failures across five power-system task categories: numerical reasoning, network topology, constraint verification, false-premise acceptance, and abstention. We evaluate ten LLMs over the benchmark, yielding 39.35\% overall correctness, with constraint verification achieving only 21.75\%, and highlighting persistent difficulties in engineering reasoning. We further investigate engineering-grounded reasoning through numerical solver support and verified network information both of which show significant performance improvement. The findings reveal important limitations of standalone LLM reasoning and motivate integrating LLMs with verified engineering tools for reliable power-system analysis.
Source:https://arxiv.org/abs/2611.22012PreviousSafe Reinforcement Learning with Hard Constraints: Policy Gradient Methods with Intersample GuaranteesNextAn Explainable Agentic AI Framework for Distribution System Load Restoration
Related posts
Introduction to CUDA Beyond the Syntax
0Hacker News New
Physics AI: The illusion of ground truth
0Hacker News New
RiddleBell Puzzle Alarm Clock
0Hacker News New
Open WebUI – Open Terminal: A Computer for Your AI Agents
0Hacker News New
LLMs abandon correct climate facts when users push back (UNESCO/UCL)
0Hacker News New
Standard Support: How AI agents ask for support
0Hacker News New