When building Identity & Access Management (IAM) systems, authentication and authorization sit directly on the hot path of every incoming request. Over the past decade—across three or four complete from-scratch rewrites—I have been refining VK Auth, an idiomatic Go IAM engine built around zero-allocation read paths, minimal memory overhead, and deterministic tail latencies.
After benchmarking both the token verification engine (controller/) and the authorization tree (graph/) across millions of users and memberships on a Ryzen 7950X, we hit an interesting systems ceiling: graph traversal became so fast that Go’s sync.RWMutex atomic reader counter now accounts for over half of saturated CPU.
Here is a breakdown of the numbers, why we explicitly rejected lock-free copy-on-write in favor of state-delta encoding (adr-017), and how we solved phased tree updates without breaking invariants.
1. Authentication (controller/): Up to 30.2M Verifications/s
In the first benchmark suite, controller/ handles token-session verification across both pure in-memory stores and a disk-backed Badger store, with and without observability enabled:
- Peak Throughput: 30.2M verifications/s at 10k users (
1.66 µsp99) and 28.1M verifications/s at 1,000,000 users (2.30 µsp99). - Disk-Backed Parity: Running 1,000,000 observed users on Badger on disk sustains 27.4M verifications/s at the exact same 2.30 µs saturated p99, while shrinking heap usage after GC from 3.2 GB down to 2.1 GB.
- Scale Headroom: Even at 2,600,000 users, saturated throughput holds at 26.1M verifications/s (
2.56 µsp99).
2. Authorization (graph/): 24.3M Answers/s & Zero Read Allocs
Unlike controller/, graph/ serves authorization queries directly from an in-memory tree backed by a publish-then-write persistence mechanism. Rather than testing toy microbenchmarks, we generated realistic organizational topologies—ranging from multinational enterprises and universities to dense chat/guild graphs:
- Zero Read-Path Allocations: Both scoped and unscoped authorization checks execute with 0 heap allocations, keeping garbage collection off the hot path.
- Target Throughput: 24.1M – 25.2M answers/s saturated across Multinational, Education, and Guild targets at a flat 2.30 µs saturated p99.
- Disk-Backed Durability: Backing the Multinational target with a 2.2 GB Badger store still answers 24.3M queries/s.
Even when pushing the tree into heavy stress ranges—scaling up to 100,611 groups (Guild) or 1,000,000 groups and 11.45M memberships (Chat)—latency degrades gracefully:
Why We Rejected Lock-Free Pointer Swaps (ADR-017)
When you vary tree depth (climbing 0 to 12 guardian levels beneath a membership) or branching factor (from 2 to 128), throughput barely moves—dropping only 4–7% across a 12-level climb.
The reason is hardware-level lock contention:
- Under saturated concurrent reads on a 16-core machine (32 threads), more than half of the CPU time on the Multinational target (and nearly 90% on the smallest trees) is spent inside
sync.RWMutex's shared reader counter—cache-line bouncing across cores—rather than traversing the tree itself. - A common reflex in Go is to eliminate
RWMutexvia lock-free atomic pointer swaps (ADR-010's original model). InADR-017, we explicitly rejected both copy-based alternatives:- Whole-tree rebuilds (
ADR-010): Building a mutated copy and swapping a pointer makes transition time—and therefore permission revocation latency—proportional to the size of the entire tree rather than the size of the change. - Structural sharing (persistent path-copying): Keeps revocation latency flat, but forces every structure (including the permission universe) into persistent data structures, adding massive comprehension and security complexity.
- Whole-tree rebuilds (
- The Solution (
In-Place Fold Under Exclusion): The tree separates canonical state (N) from a pending delta (N+1 - N). Writers compose mutations into the delta without blocking readers, then take the write lock only to fold the delta into canonical state in place. Revocation latency scales strictly with the size of the delta, and Go's race detector can verify correctness by inspection. - Why We Accepted the
28M/sCeiling (§6): BecauseRLockcontends on a shared counter, throughput caps around 28M answers/s regardless of core count. InADR-017 §6, we explicitly state and accept this limit because it sits ~480× above plausible enterprise peak demand (50,000checks/s)—and any deployment exceeding it can shard the hierarchy (ADR-016).
Phased Writes Without Breaking Invariants
The hardest engineering challenge wasn't read speed—it was updating the authorization tree safely while millions of reads are in flight.
Moving from single atomic writes to a phased write mechanism introduces irreducible complexity: multiple tree invariants must remain intact across intermediate states so concurrent readers never observe torn permissions. Combined with state-delta encoding, this yields atomic visibility in the best case and strict eventual consistency in the worst—without ever violating security invariants.
Engineering With Intent (Preventing AI Refactor Loops)
Retrofitting a complex mechanism like phased writes onto a codebase that has evolved across multiple rewrites requires strict architectural guardrails. I structured the workspace around Vigil Keep (ibe-workflow-template)—an Intent-Based Engineering (IBE) workflow combining structured context management and explicit Architecture Decision Records (ADRs).
When using AI coding agents on irreducible systems problems, letting the agent iterate blindly at the code level causes context confusion and endless loops. Instead, we enforce a strict directional ladder (Influence drives down; feedback rises only as high as necessary):
- Forward Drive:
Intent → Decisions (ADRs) → Design / Specs → Implementation - Backward Validation: If implementation fails against intent, feedback steps up only to the first rung capable of resolving the missing constraint (Implementation → Specs → Decisions → Intent).
- Forward Remediation: Once the constraint or trade-off is locked in the ADR/Spec (such as
adr-017), execution drives down once and concludes—or escalates to a human if intents conflict.
(You can see a CSS diagram of how influence and feedback move across these rungs on Noctranet's Approach page.)
Explore the Codebase
VK Auth is open-source under Apache 2.0 and split into focused modules:
-
graph/— In-memory authorization tree, zero-alloc traversal, and state-delta phased writes -
controller/— Token and session verification engine -
register/— Identity and credential management -
workspace/— Reproducible benchmark generators, stress suites, andPERFORMANCE.md
Check out the repositories and ADRs on GitLab: https://gitlab.com/vigilkeep/auth
If you’ve weighed sync.RWMutex + state-delta mutations against copy-on-write structures in high-throughput Go services, I’d love to compare notes in the comments!



