AI & ML
Beyond the Leaderboard: Multi-Dimensional Evaluation of Dense and Mixture-of-Experts Models for Automated Program Repair
arXiv cs.SE0 views
arXiv:2610.08173v1 Announce Type: new Abstract: Automated Program Repair (APR) with language models is usually evaluated by whether a generated patch passes the test suite, which can hide differences in maintainability, security, and computational cost. We propose a Weighted Quality Index (QI), inspired by the ISO/IEC 25010 software quality model, that combines functional correctness, maintainability, security, and generation efficiency under configurable weighting schemes. We evaluate three dense Qwen2.5-Coder models (3B, 7B, 14B) and the 16B-parameter DeepSeek-Coder-V2-Lite Mixture-of-Experts (MoE) model (2.4B active parameters) on 40 QuixBugs and 90 Defects4J bugs, all run locally on identical hardware to control for infrastructure effects. Model rankings change with the weighting scheme, showing that single-metric evaluation can hide trade-offs. The MoE model shows almost no statistically significant difference in correctness from the 7B and 14B dense models (McNemar's exact test) while using 3-6 times fewer active parameters, whereas correctness increases significantly across the three dense scales. These results suggest that active parameter count can be a more informative lens than total parameter count for sparse code models.
Source:https://arxiv.org/abs/2610.08173PreviousHarness Engineering for Software Engineering via Modular Executable Dev-PrimitivesNextNewer and Bigger, but Safer? A Longitudinal Study of the Functionality-Security Gap in LLM-Generated Code
Related posts
Google made $63 billion from Search. So why are you getting fewer clicks?
0dev.to
What a GitHub Actions job costs on each runner: CI Bench v1 cost per job by workload
0dev.to
Ballot Rejections Have Soared in Rural Counties as USPS Reduces Mail Service
0Hacker News
Cloudflare DNS Zone Builder cuts peak memory 85% and halves build delay
0dev.to News
O dia em que a cliente pediu desconto num bolo de pistache (e pagou R$ 30 num cafe)
0dev.to Design
O telefone tocou as 19h: Por que cancelam seu contrato primeiro na crise?
0dev.to Design