AI & ML
A Validated Dataset and Benchmark for Coherent Multi-Diagram SysML Models
arXiv cs.SE0 views
arXiv:2610.07356v1 Announce Type: new Abstract: Systems engineers use several diagrams to describe the structure and behavior of systems. Engineers create these diagrams together to make sure that they use the same elements and remain consistent with one another. Large language models can generate diagrams as text or code, which makes it possible to create system diagrams automatically. However, their ability to generate coherent sets of diagrams is not well understood, and existing datasets and benchmarks do not directly measure this ability at scale. We introduce SEMAADB (Systems Engineering Modeling Assistant with AI Dataset and Benchmark), a dataset of 3,000 engineering contexts and 15,000 diagrams. Each context contains five connected SysML views: Requirement, Block Definition, Activity, State Machine, and Sequence. Here, a view is a diagram that presents one aspect of a system. We checked the diagram sets for consistency and valid rendering. A set of 100 contexts is also human-verified and forms the benchmark test set. We evaluate three language models on two tasks. In diagram repair, the strongest model repairs 64.3% of semantic errors . In cross-diagram update the best propagation F1 is 80.7% when a model applies one change across related diagrams. The results show that syntax repair is nearly solved, but semantic repair and consistency across diagrams are still challenging tasks for models. SEMAADB therefore provides both a large diagram resource and a set of benchmarks for measuring coherent multi-diagram SysML generation.
Source:https://arxiv.org/abs/2610.07356PreviousCatching Developers in the Flow: Low-Latency Agentic Program Repair at Google ScaleNextCogAdapt: Cognition-informed Sparse Adaptation of Code LLMs
Related posts
How to Turn a Process Document Into a Working App With AI
0dev.to Machine Learning
Which AI subscription should I buy?
0dev.to Machine Learning
Your Classifier Says 0.97. Six Ways That Number Lies in .NET 10
0dev.to Machine Learning
Which AI song generator fits a lyrics-first workflow?
0dev.to Tutorial
Anima 2.9B in ComfyUI: Six Workflows, From Official T2I to Tiled Redraw and Smart Img2Img
0dev.to Tutorial
How to calculate how many cartons fit on a pallet (with a worked example)
0dev.to Tutorial