Database
A Multi-Domain Study on the Effectiveness of LLMs at Writing Property-Based Tests
arXiv cs.SE0 views
arXiv:2610.34487v1 Announce Type: new Abstract: A property-based test (PBT) checks a property of a program on many generated inputs, but writing one takes a property, a generator of valid inputs, and an oracle. LLM-based PBT writers produce all three, yet they have been evaluated only on Python libraries. We study how two of them, Proptest-AI and Agentic-PBT, generalize to ten programs in five domains: deep learning libraries, databases, system software, compilers, and LLM serving engines. Most of these programs take text in their own language, such as queries, configurations, and programs. Agentic-PBT covers more branches than Proptest-AI on seven of the ten programs with Opus 5 and on all ten with GLM-5.3-Flash. Domain-specific fuzzers cover more of the compilers and of PostgreSQL and less of PyTorch than Agentic-PBT. We filed 180 bug reports, 44 of which duplicate reports that others filed after the models' knowledge cutoffs; maintainers have confirmed 65 of the others, in all five domains, and fixed 23. We draw lessons for building and evaluating such writers: low observability hurts them, and mature fuzzers paired with LLM-written properties could check that each feature behaves as documented.
Source:https://arxiv.org/abs/2610.34487PreviousSymLift: Neural Semantic Lifting for Compile-Free Vulnerability Detection in Context-Incomplete Code FragmentsNextMultimodal Requirements Traceability Link Recovery in Industrial Settings
Related posts
Goravel feature testing for a REST API
0dev.to Testing

How to find and fix slow PostgreSQL queries
0dev.to Database
Goravel models and migrations with Postgres
0dev.to Database
Goravel belongs to relationships and foreign keys
0dev.to Database
Goravel has many relationships and N+1 queries
0dev.to Database
Goravel factories and seeders with fake data
0dev.to Database