AI & ML
Eliciting Specifications: When Does Interactive Clarification Help?
arXiv cs.SE0 views
arXiv:2610.22382v1 Announce Type: new Abstract: LLMs can generate Lean specifications from task descriptions, but those descriptions are often underspecified or ambiguous and may omit verifier-relevant edge cases, yielding imperfect formal specifications. We study whether an elicitation agent (EA) can improve verifier-checked Lean specifications by seeking clarification from a reference-grounded simulated developer before formalization. The agent decomposes the task, identifies unresolved decisions, asks bounded clarification questions, and uses the answers to write a refined natural-language specification that an LLM formalizer then translates into Lean. On 189 Verina Lean 4 specification-generation tasks, we compare single-shot generation with elicitation under question budgets of k=3 and k=10, allowing at most three or ten EA questions per task, across same-model and cross-model configurations. The same-model Claude Sonnet 4.6 pipeline has negative strict pass@1 estimates at both budgets, showing elicitation is not a uniform accuracy booster. In contrast, the Sonnet 4.6-to-GPT-5.6 and Sonnet 4.6-to-Gemini 3.8 Flash pipelines have positive strict estimates. The largest strict result reaches 65.6%, a +6.3-point change (+12 tasks), in the Sonnet 4.6-to-GPT-5.6 pipeline. Six of eight conditions have positive compile-conditional estimates, with a maximum cross-model difference of +4.9 points among compiling outputs. Treatment estimates also differ across the basic and advanced splits; the largest advanced-task change occurs in the Sonnet 4.6-to-GPT-5.6 configuration at k=10, from 42.0% to 50.6%. Failure analysis identifies postconditions as the largest source of degradation. These findings suggest future autoformalization systems should treat elicited natural-language specifications as intermediate artifacts and optimize jointly for semantic correctness among compiling outputs and reliable translation of clarified information into Lean.
Source:https://arxiv.org/abs/2610.22382PreviousComputing Extinction Barriers in a Quorum-Sensing Reaction NetworkNextEnterpriseSynth: Zero-Execution SFT and Evaluation Data from OpenAPI Specs
Related posts
7.9% to 48.1%: LLM rewrote Qwen weights, no training
0dev.to
Spousal Open Work Permits in Canada: Who Qualifies in 2026 and What to Prepare
0dev.to
Reading ingredient lists from phone photos: a crop bug, a model bench, and why the AI never gives the verdict
0dev.to
Edital exige guardrail
0dev.to
Convert an SVG to a PNG in the Browser, Without Uploading the File
0dev.to
Generative Simulation Benchmarking for planetary geology survey missions for extreme data sparsity scenarios
0dev.to