When you first learn concurrent programming in Rust, the std::"thread"::spawn function feels like magic. Got 10,000 URLs to parse? Just wrap the scraper function in a loop, spawn 10,000 threads, and let the operating system figure it out.
But as your systems scale, that magic quickly turns into a nightmare.
Spawning more OS threads than you have physical CPU cores forces the Linux kernel into heavy context switching. The OS frantically pauses and resumes threads, thrashing your CPU caches and wasting precious clock cycles just figuring out who gets to run next.
To achieve maximum throughput for a high-performance Web-to-Markdown extraction engine I am building, I needed a system that keeps the CPU cores pinned at 100% execution, bypassing the OS scheduler entirely.
Here is how I built a 1:1 hardware-mapped, lock-free, work-stealing job scheduler from scratch in Rust using crossbeam-deque.
The Architecture: Silicon Over Syscalls
The goal of a custom scheduler is to spawn exactly as many OS threads as you have logical CPU cores (e.g., 8 cores = 8 threads). These threads boot up once, live forever, and constantly hunt for tasks.
But how do you distribute the work safely without wrapping everything in a massive, slow Mutex? We use a Chase-Lev work-stealing deque.
Instead of fighting over a single locked queue, every thread gets its own private queue.
The thread operates on the bottom of its own queue. This requires zero locks. If a thread runs out of work, it becomes a "thief". It uses hardware-level Compare-And-Swap (CAS) atomic instructions to rip jobs from the top of a peer's queue. No system calls, no OS intervention—just pure silicon arbitration. Step 1: Mapping the Memory Boundaries
We need to define three layers of state: the public API, the synchronization primitives, and the internal worker context.
use crossbeam_deque::{Injector, Stealer, Worker, Steal}; use std::sync::{Arc, Mutex, Condvar}; use std::thread::JoinHandle; // The payload: a heap-allocated closure that owns its data and is thread-safe pub type Job = Box<dyn FnOnce() + Send + 'static>; // The Intercom: Used to wake up starving threads pub(crate) struct SleepSignal { keep_running: Mutex<bool>, cvar: Condvar, } // The Manager: Lives on the Main Thread pub struct JobStealer { global_queue: Arc<Injector<Job>>, sleep_signal: Arc<SleepSignal>, workers: Vec<JoinHandle<()>>, } // The Worker Brain: Moved permanently into the OS Thread struct WorkerContext { id: usize, local_deque: Worker<Job>, stealers: Vec<Stealer<Job>>, global_deque: Arc<Injector<Job>>, sleep_signal: Arc<SleepSignal>, } Notice the separation of concerns: The JobStealer manager holds the handles to shut the system down, while the WorkerContext holds the Stealer keys to access its peers.
Step 2: The Execution Loop (The Hunt)
Once we spawn the OS threads and move the WorkerContext into them, they get trapped in an infinite loop {}. The threads execute a strict hierarchy of operations to find work, optimizing for cache locality.
impl WorkerContext { fn run(self) { loop { // Priority 1: Check the local, lock-free queue if let Some(job) = self.local_deque.pop() { job(); continue; } // Priority 2: Global Batch Steal // Take a chunk of jobs, put them in local, return one to run now if let Steal::Success(job) = self.global_deque.steal_batch_and_pop(&self.local_deque) { job(); continue; } // Priority 3: Peer Stealing (Hardware Atomics) let mut stole_job = false; for stealer in &self.stealers { if let Steal::Success(job) = stealer.steal_batch_and_pop(&self.local_deque) { stole_job = true; job(); break; } } if stole_job { continue; } // Priority 4: All queues are empty. Time to sleep. self.sleep(); } } } Step 3: Navigating the Sleep Trap
If all queues are empty, a thread must yield its CPU cycle back to the OS, otherwise it will burn 100% CPU spinning in a loop {}.
We use a POSIX Condvar to sleep. But sleeping in systems programming comes with two notorious traps:
The Lost Wakeup: If the Main Thread signals new work a microsecond before the Worker goes to sleep, the Worker will miss the signal and sleep forever. We fix this by tying the Condvar to a Mutex. It makes the state-check and the sleep-command one unbreakable atomic transaction. The Spurious Wakeup: The Linux scheduler can randomly wake up parked threads even if no one pressed the alarm. We fix this by putting the sleep command inside a while loop. impl WorkerContext { fn run(self) { loop { // Priority 1: Check the local, lock-free queue if let Some(job) = self.local_deque.pop() { job(); continue; } // Priority 2: Global Batch Steal // Take a chunk of jobs, put them in local, return one to run now if let Steal::Success(job) = self.global_deque.steal_batch_and_pop(&self.local_deque) { job(); continue; } // Priority 3: Peer Stealing (Hardware Atomics) let mut stole_job = false; for stealer in &self.stealers { if let Steal::Success(job) = stealer.steal_batch_and_pop(&self.local_deque) { stole_job = true; job(); break; } } if stole_job { continue; } // Priority 4: All queues are empty. Time to sleep. let mut keep_running_guard = self.sleep_signal.keep_running.lock().unwrap(); // The while loop protects against Spurious Wakeups while *keep_running_guard && self.global_deque.is_empty() { // wait() safely hands the Mutex key to the OS, goes to sleep, // and catches the new key when it wakes up. keep_running_guard = self.sleep_signal.cvar.wait(keep_running_guard).unwrap(); } // If we woke up because the factory is closing, shatter the loop if !*keep_running_guard { std::process::exit(0); // In actual code, we break the outer loop } } } Step 4: The Graceful Shutdown
If the main program finishes and exits, the OS will violently murder the sleeping worker threads, potentially corrupting memory. We implement Rust's Drop trait on the manager to politely fire the workers instead.
impl Drop for JobStealer { fn drop(&mut self) { { let mut keep_running = self.sleep_signal.keep_running.lock().unwrap(); *keep_running = false; self.sleep_signal.cvar.notify_all(); // BLAST THE FIRE ALARM } // Mutex auto-unlocks here // Wait for threads to cleanly pack their desks let handles = std::mem::take(&mut self.workers); for worker in handles { worker.join().unwrap(); } } } By flipping the boolean and calling notify_all(), the kernel makes all sleeping threads awake. They see the boolean is false, break out of their infinite loops, and peacefully return their memory to the OS.
The Benchmark: 1 Million Jobs
To prove the efficiency of bypassing the OS scheduler, I wrote a benchmark that blasts 1,000,000 CPU-bound tasks into the Global Injector.
Running this in --release mode on my machine with a 4-core processor yields incredible throughput:
Bash
==================================================
BENCHMARK COMPLETE
Total Execution Time : 2.450011653s
Throughput : 408161.32 jobs / second
Instead of the Linux kernel drowning in 1,000,000 thread context switches, exactly 4 threads are spun up. They ruthlessly steal batches of work from the global queue and each other, relying entirely on L1/L2 CPU caches and silicon atomic instructions to synchronize.
Conclusion
This scheduler will serve as the concurrency backend for my Rust-based Web-to-Markdown extraction engine, allowing it to parse thousands of DOM trees across multiple cores without dropping a single frame.
The full source code for this scheduler is available in my systems_engineer workspace on GitHub https://github.com/14elias/zero-copy-protocol/tree/main/thread-pool.
Understanding how to construct your own thread pool forces you to learn what happens when your code crosses the boundary into the operating system.


