TL;DR

Stop burning tokens and introducing multi-second cloud latency just to make structured routing decisions. laya-mlx is a native Apple MLX runtime for Laya typed decision models that clocks in at an astonishing 7–14 ms on an M3 Max. By stripping away autoregressive text generation and heavy PyTorch dependencies, it gives local AI workflows instant, deterministic decision-making entirely on-device.

Key Features & Benchmarks

  • Blazing Fast (7–14 ms on M3 Max): Executes structured, typed decisions at real-time speeds, unlocking ultra-responsive local agent loops.
  • Pure MLX Engine: Built directly on Apple's unified memory architecture and Metal performance shaders—completely eliminating PyTorch overhead.
  • Zero Generative Waste: Avoids token-by-token generation overhead; focuses purely on strict typed outputs, classification, and branching logic.
  • Air-Gapped & Deterministic: No cloud network hops, token fees, or API rate limits. Your decisions remain strictly private on edge hardware.
  • Minimal Dependency Tree: Lightweight Python integration that fits seamlessly into modern macOS developer toolchains.

Quick Start

Get up and running with laya-mlx in just a few lines of code:

pip install mlx laya-mlx 
from laya_mlx import LayaDecisionModel # Load a pre-trained typed decision model natively into Apple Silicon unified memory model = LayaDecisionModel.from_pretrained("mizore/laya-decision-v1") # Provide input context and define the decision target context = "User requested database query optimization on production cluster." decision = model.decide( context=context, schema=["escalate_to_dba", "run_auto_explain", "reject_request"] ) print(f"Decision: {decision.action} (Latency: {decision.latency_ms:.2f}ms)") # Output: Decision: run_auto_explain (Latency: 9.42ms)

Why It Matters

Most modern agentic architectures overuse massive 7B+ LLMs for tasks that are fundamentally multi-class classifications or rigid tool routers. Invoking an LLM via cloud API adds 500–1500 ms of latency and burns cash; running a 7B model locally via Ollama or vLLM consumes gigabytes of VRAM and still takes hundreds of milliseconds.

laya-mlx is built for engineers building autonomous local agent systems, real-time code assistants, and edge devices where sub-20ms routing is mandatory. If you need reliable, typed branch logic without spinning up a heavy generative pipeline, this is the architecture to watch.

Call To Action

Are you optimizing your local AI infrastructure for speed and efficiency? Star the project on GitHub: mizorewww/laya-mlx.

Follow 'Local AI & Infra Daily' for daily deep dives into the fastest runtimes, local model optimizations, and open-source AI infrastructure!