IDKLETSCODESOMETHIN / VeloGrad
Bare-metal, native-first deep learning & autoregressive inference engine in C++20, Rust, CUDA & Python.
VeloGrad (Beta)
VeloGrad is a bare-metal, native-first, pure-compiled deep learning and autoregressive inference engine engineered from scratch in C++20, Rust, CUDA, and Python by Ayushman Singh Rathore.
It eliminates Python runtime overhead, framework memory bloat, and garbage collection pauses by introducing an aligned static arena memory architecture, ahead-of-time (AOT) fused compute kernels, an empirical ~66.7% KV-cache and activation VRAM reduction engine, enterprise ZeRO-3 / FSDP distributed scaling with CPU/NVMe offload, and a natural-language autonomous training orchestrator (velograd.instruct).
Important
Public Beta Notice (v1.1.0-beta)
VeloGrad v1.1.0-beta is now in public beta.
All core compute kernels, hardware-fused operators, universal frontier model architectures (from 100M edge models up to 1T+ parameter MoE networks), Paged KV-cache VRAM reduction, ZeRO-3 FSDP parameter sharding, and natural-language autonomous training directives (velograd.instruct) have been verified with complete zero-defect automated test and benchmark suites. Community testing, empirical benchmarks, and feedback are…
Hey DEV community! My name is Ayushman Singh Rathore, and I’ve been building VeloGrad—an open-source tensor, autograd, and distributed training framework written in C++20, Rust, and CUDA with Python bindings.
Today I’m releasing v1.1.0-beta into Public Beta. Before diving into the architecture, I want to share an honest builder lesson from v1.0.0 that completely changed how I ship code.
The v1.0 Wake-Up Call: Auditing AI-Generated Scaffolding
When I first pushed v1.0.0, I leaned too heavily on an AI coding agent to draft the initial scaffolding and README.md. Right after publishing, I audited the repository line by line and realized the agent had hallucinated benchmark claims and left broken Python bindings and stubbed functions.
Instead of leaving an unverified repo up, I pulled back, stripped out every unverified claim, added a clear Public Beta status, and spent days rewriting, compiling, and testing the actual systems code so that every single line in the README is backed by real, executable code.
What’s Inside VeloGrad v1.1.0-beta
VeloGrad is designed to strip away Python dispatch overhead and GPU memory fragmentation during model training and fine-tuning by keeping the memory arena, autograd tape, and fused kernels inside a tight native runtime—while automatically falling back to AVX2/NEON SIMD kernels on CPU-only Linux, macOS, and Windows machines.
1. Fused Forward & Backward Kernels (CUDA + SIMD Fallback)
Instead of launching separate memory-bound kernels for every operation, VeloGrad fuses common transformer primitives into single passes:
Fused GEMM + Bias + Activation (Forward & Backward)
Fused RMSNorm + Residual Add
Fused RoPE + Tiled Flash-Style Self-Attention
Single-Pass Fused Cross-Entropy Loss (computes log-softmax and gradients without materializing massive [batch, seq, vocab] logit tensors in VRAM)
Fused Multi-Tensor AdamW
2. Static Arena Allocation & ~66.7% KV/VRAM Compression
Dynamic allocations (cudaMalloc) during training loops cause fragmentation and latency spikes. VeloGrad pre-allocates a deterministic static memory arena and pairs block-quantized Paged KV-Cache with activation checkpointing (recompute_backward), cutting KV-cache and activation VRAM overhead by up to ~66.7% compared to unquantized baselines.
3. ZeRO-3 Sharding + CPU Parameter Offloading
To let developers train and fine-tune larger models on consumer GPUs without hitting OOM walls, VeloGrad implements ZeRO-3 (FSDP) parameter sharding and CPU parameter offloading over NCCL (with a cross-platform TCP/Ring fallback when NCCL isn't present), supporting SafeTensors and GGUF weights across Llama, Qwen, DeepSeek, Gemma, and Mistral model families.
4. Natural-Language Training Orchestration (velograd.instruct)
Alongside standard Python Tensor and nn.Module APIs, v1.1.0-beta introduces velograd.instruct, which auto-detects your local GPU/CPU topology and maps plain-English training directives directly into a configured sharding and training pipeline:
import velograd
Auto-detects available GPUs/RAM, loads weights, and configures sharding + VRAM compression
velograd.instruct(
"Fine-tune Llama-3.2-3B on local_dataset.jsonl with ZeRO-3 offloading and 66.7% KV compression"
)
How to Test v1.1.0-beta on Your Machine
If you want to compile it and run the reproducible PyTorch comparison and VRAM profiling scripts on your own GPU or CPU laptop:
git clone https://github.com/IDKLETSCODESOMETHIN/VeloGrad.git
cd VeloGrad
pip install -e .
Run the verification suite and benchmarks
pytest
python benchmarks/bench_vs_pytorch.py
python benchmarks/profile_vram.py
What’s Next & Call for Beta Testers
Because this is a Public Beta, real-world hardware diversity is what matters most right now. If you test VeloGrad on NVIDIA RTX cards, multi-GPU workstations, Apple Silicon, or Windows/Linux CPUs, please open an issue on GitHub with your build logs or benchmark numbers!
Repository: https://github.com/IDKLETSCODESOMETHIN/VeloGrad


