AI & ML
Does More Thinking Pay Off? Reasoning Cost and Correctness of LLMs in Code Generation and Bug Fixing
arXiv cs.SE0 views
arXiv:2610.36952v1 Announce Type: new Abstract: Many LLMs now emit an explicit reasoning trace before they write code, but benchmark results for code generation and bug fixing usually report correctness alone. It is therefore unclear how much the extra reasoning contributes relative to the tokens it consumes. We examine this question with 866 Python tasks drawn from HumanEval Pro, MBPP Pro, and BugsInPy under five model settings. We look at the accuracy gain and token overhead of reasoning, the behaviors that appear inside reasoning traces, and whether a shortened trace still leads to a correct solution. Turning reasoning on in Qwen3-32B over five runs improves mean pass@1 on HumanEval Pro from 71.1% to 76.5% and mean tests pass on BugsInPy from 3.5% to 7.9%. On MBPP Pro the improvement from 68.1% to 69.5% is not statistically significant, even though the model uses a median of over 2,000 thinking tokens per task. Across models, a longer median trace does not reliably mean higher accuracy. A qualitative analysis shows that traces mostly choose and check an implementation in code generation, while in bug fixing they mostly locate the fault and plan the repair. We then truncate stored traces offline and ask Qwen3-32B with reasoning off to produce a solution from what remains. Retaining the first half of a trace yields no statistically significant drop in correctness. Stopping when the trace becomes repetitive seldom triggers, and stopping once solution code first appears benefits some models but harms others. Neither rule beats plain fixed truncation. These results suggest that more reasoning is not always better and that studies should report reasoning cost alongside correctness.
Source:https://arxiv.org/abs/2610.36952PreviousBeyond QASecClaw: Agent Scaling, Verifier Feedback, and Evidence Reliability in Software AssuranceNextToward Circuit-Aware Mutation Testing for Quantum Programs: Exact Visibility and Execution Planning
Related posts
Best Dyson Vacuum Deals (2026): Dyson Spot+Scrub Ai, Dyson V16 Piston Animal
0Wired
CPU, GPU, TPU, NPU: Who Will Power the Future of Computing?
0dev.to

Lower loss, worse stories: small NCMs trained on TinyStories
0dev.to

I Built LLM Explained: 43 Interactive Pages That Take You From Tokens to AI Agents
0dev.to OpenSource
MIT CSAIL: Astra edita 0,016% de pesos de Qwen3.5-0.8B y sube MATH-500 a 48,1%
0dev.to Programming

Gemma 4 in Rust on TorchTPU: A Native TPU Rig With PyO3 and MCP
0dev.to Rust