AI & ML
torchtest2: One-Call Sanity Checks for PyTorch Models,Evaluated on Real and Unseen Architectures
arXiv cs.SE0 views
arXiv:2611.23273v1 Announce Type: new Abstract: Many bugs in deep learning code are silent: a layer that is never called, an optimizer created before the classification head was replaced, a recurrent layer that treats the batch as time, or an ignored attention mask. The code runs, the loss decreases somewhat, and the defect surfaces, if ever, after days of GPU time. We present torchtest2, an open-source library that runs 18 sanity checks on a single batch with one call, restores the PyTorch state it touches, and adapts to model families through plugins. We evaluate it by mutation analysis with 17 classes of silent bugs on 103 models from torchvision, timm and Hugging Face transformers, on 600 random architectures and on 20 hand-written realistic bugs, against the checks of the original torchtest, overfitting a single batch, a smoke test and a training monitor in the style of AUTOTRAINER and DeepDiagnosis. Every mutant is compared with its healthy counterpart, and intervals come from a bootstrap over models. After freezing the tool, we ran it on 191 models of architecture families it had never seen: counting, for every tool alike, a failing report where the healthy model's report does not fail, it detected 78% of the injected bugs (95% CI 75-80), against 40% for the original torchtest checks and 48% for the training monitor. On real models its batch-independence check found a bug in timm's GCViT that couples the images of a batch. Real models also exposed false alarms that neither the tool's own test suite nor random architectures had revealed. Two independent collections of real silent bugs mark the limits of the approach: only 3 of 28 reproduced PyTorch bugs from Stack Overflow and 3 of the 18 silent training errors released with TrainCheck are within reach of a pre-training model tester. A median check takes 1.0 s. We release the library, the benchmark and all raw results.
Source:https://arxiv.org/abs/2611.23273PreviousSkip-Level Meetings: Benefits, Risks, and Design Considerations Informed by Bypassed Managers' ExperiencesNextA Concurrent Structured Record Mirrored the Prose It Accompanied: A Pre-Registered Conceptual Replication and Extension of the Hide Pending Tool Call Scenario
Related posts
Im looking fr any free ai models / tools are there?, leaVE UR Lights💡
0dev.to
I checked 10 SEO agency websites for AI-search readiness. The real bugs were the boring ones.
0dev.to

How much GPU memory does an LLM need? The whole sum, worked out on real cards
0dev.to

Routing Upscaling Jobs by Output Size, Not Input Size
0dev.to
A ten-second Flutter pause with a visible Skip button
0dev.to UX

Thống kê lịch sử đối đầu Aston Villa - Sunderland: Đại diện Midlands áp đảo toàn diện
0dev.to Data