Backend
The Complexity Kink: A Prompt-Side Structural Complexity Index for Code-Generation Reliability
arXiv cs.SE0 views
arXiv:2609.19616v4 Announce Type: replace-cross Abstract: Measuring complexity from a model's own code is biased by failure: a hard prompt can produce a short, broken program that looks simple. We instead score each prompt's structure on six dimensions from the prompt alone, never from generated code or test results. We draw 5,000 Python prompts across six bands of a preliminary single-rater score, and four out-of-panel LLM judges rescore every prompt (composite ICC(2,1) = 0.872). Twenty-one models answer every prompt (105,000 generations), and an independent LLM auditor, checked against known-answer cases, reviews each unit-test verdict. Averaged over models, pass rate has a nonmonotone breakpoint at composite 14.0: 83.4% at or below, 90.1% above. That jump is not a law of difficulty: with task-type fixed effects the breakpoint stays at 14.0 but the adjusted jump shrinks from 6.8 to 0.4 points, and neither of the benchmark's two prompt sources reproduces it on its own. Nineteen models pass more often above their own breakpoint and two less often; one of those declines comes from API calls that returned nothing. In post hoc follow-up, Kimi K2.5 shows the clearest decline, at 14.25, under a prompt-source control and within the later source alone. Among failed generations with measurable output complexity, 25.0% pair a prompt composite above 8 with output complexity of at most 10, twice the share among passing ones. The pooled breakpoint moves to 11.25 on raw test outcomes. Human agreement with the index is moderate, paraphrase and cross-language rescoring preserve its ordering, and overidentification tests reject the six dimensions as instruments, so we make no causal claim. The contribution is a prompt-side way to locate where each model's pass rate changes and to test how much that depends on the prompt mix and on how success is scored.
Source:https://arxiv.org/abs/2609.19616PreviousWhen Models Edit Too Much: On the Fidelity of Minimal Code EditsNextCramer-Rao Type Bounds for Generalized Exponential Families
Related posts
How to bulk rename files safely on Windows, macOS and Linux
0dev.to
Extract Audio From a Video in the Browser: WAV or WebM Opus, Checked Byte by Byte
0dev.to

One unrecognized selector killed Messages.app 15 times
0dev.to
ShengChang: Import a Novel Chapter, Get a Mixed Radio-Drama Episode
0dev.to
Why Every Backend Needs Multi-Tier Caching (L1 RAM + L2 Redis)
0dev.to
SFXMint vs Freesound vs Kenney: Free SFX for Games, Video, and AI Agents
0dev.to