AI & ML
A Comprehensive Evaluation Framework for Conversational Home Energy Management Systems
arXiv cs.HC0 views
arXiv:2610.00073v2 Announce Type: replace Abstract: The growing complexity in home energy management (HEM) demands advanced systems that guide occupants toward informed energy decisions reflecting their background, preferences, and context. Large language model (LLM)-integrated HEM systems (HEMS) have demonstrated promise, but previous studies relied on single-turn or single-task evaluations with response accuracy as the primary metric. Whether such systems deliver effective interactions across the extended multi-turn dialogues typical of real-world use remains an open question. This study introduces a comprehensive evaluation framework of LLM-integrated HEMS derived from the Goal-Question-Metric methodology, organized across five categories: task performance, factual accuracy, interaction quality, control capability, and system efficiency. A total of 23 metrics across multi-turn conversations are proposed and an LLM-as-judge pipeline is employed to enable scalable automated scoring. Its reliability is validated against three trained human coders: after iterative rubric calibration, twelve of the fifteen LLM-scored metrics reached strong agreement (ICC >= 0.73), three of them perfect, while the remaining three exhibited near-zero variance in human scores and are instead reported via mean absolute error (0.04-0.28). To demonstrate the framework's effectiveness, 970 dialogues -- 16 scenarios and five personas -- were generated and evaluated across four conversational HEMS configurations spanning a sophistication gradient, from a vanilla LLM with raw energy data to a multi-agent HEMS. The framework distinguished the four configurations across multiple evaluation dimensions, revealing their respective strengths and weaknesses. This study contributes to conversational HEMS by providing a reproducible, multi-dimensional evaluation methodology that comprehensively assesses sustained, context-aware system performance.
Source:https://arxiv.org/abs/2610.00073PreviousProceedings of The First Reflection in Creative Experience (RiCE) WorkshopNextAI Agents Push Humans Out of the Loop
Related posts
They Found You Through a Photograph…
0dev.to
The Joy of Boring Code: Why Writing "Clever" Code is Overrated
0dev.to
From fMRI to images: what Brain-IT reconstructs—and what it does not
0dev.to
Vad ska vi automatisera först? En poängmodell för småföretag
0dev.to

Octopus Energy is bringing its sold-out plug-in solar panels to UK small businesses, says move was a 'no-brainer' following huge demand
0TechRadar
LVMH’s Media Division announces strategic partnership with OpenAI
0OpenAI Blog