Porting Concurrent C++ to Rust: Field Notes From a Few Migrations

Who should read this !

If you are a Rust engineer who has just been handed a concurrent C++ codebase — a network service, an ingestion daemon, a gateway, a media pipeline — and told "we're moving this to Rust", this is the post I wish someone had handed me the first time it happened to me.

A caveat before anything else: this is not the story of one migration. It's a composite of a few I've done or consulted on over the last several years. The systems were different (an edge proxy, a log ingestion daemon, something broker-shaped), the teams were different, and the outcomes were different. The incidents I describe happened, but I've deliberately blurred which codebase each one came from, and I've rounded the numbers. Where I generalise, treat it as pattern-matching over a small sample, not a law.

The thesis I'll keep coming back to is this: in a concurrent C++ codebase, the concurrency is the migration. Types port mechanically. Serialization ports mechanically. Business logic is tedious, but it is only tedious. The thing that eats your schedule, your sleep, and your relationship with the C++ engineers who wrote the original is the layer where threads, locks, io_contexts and atomics meet. That's where the implicit contracts live — "this handler never runs concurrently with that one", "this queue has exactly one producer", "this object is never destroyed while that thread is alive" — and Rust refuses to compile until every one of them is made explicit. That is the whole reason you're moving. It is also why it hurts.

The codebases I'm describing used three concurrency styles, usually all at once and often in the same file:

• classic std::thread + std::mutex + std::condition_variable, with a hand-rolled thread pool or two
• Boost.ASIO with completion handlers, and in the newer parts, C++20 coroutines on asio::awaitable
• a small number of hand-written lock-free structures on the hot paths: an SPSC ring, an MPSC queue, a seqlock, a few std::atomic flags doing load-bearing work

If your codebase looks like that, read on. If it's OpenMP/TBB numeric code, the audit and testing sections still apply, but the Rust side is rayon, not Tokio, and I don't cover it here.

I'll give both migration strategies — incremental via FFI, and the big-bang rewrite — equal time, because I've done both and I don't think either is the default. Which one is right depends mostly on whether the concurrency architecture you inherited is something you want to keep.

Step zero: audit the concurrency before you write any Rust

Every migration I've been part of that went badly skipped this step. The ones that went well spent two to four weeks on it before a single cargo new.

The output is a document, a boring one, that answers: what threads exist, who creates them, what they own, what they share, and how they talk. In C++ none of this is in the type system, so you have to find it by reading. The point of writing it down is that Rust is going to ask you every one of these questions at compile time anyway. You'd rather have the answers ready than discover them one E0277 at a time.

What I look for, concretely:

Thread inventory. Grep for std::thread, pthread_create, boost::thread, std::async, and every io_context::run. For each: when is it spawned, when (if ever) is it joined, and what is its lifetime relative to the object that spawned it. The "detached thread that references a member of the object that spawned it" pattern has shown up in every codebase I've audited. Each one of these is a Rust compile error, pre-announced, because std::thread::spawn demands 'static.

Lock inventory. Every std::mutex, shared_mutex, recursive_mutex, and homemade spinlock. For each: what data does it actually protect — not what the comment says, the comment is from 2017 — and what is its lock-order relationship with the others. Draw the lock-order graph on a whiteboard. If there's a cycle, you've found a latent deadlock that hasn't fired yet because of timing. Flag every recursive_mutex specially: Rust's std has no equivalent, and its presence almost always means re-entrancy the design never intended.

Condition variables. For each condition_variable, find the predicate, every notify site, and whether the wait is in a loop or uses the predicate overload. Spurious wakeups translate fine. What I'm really hunting is the lost wakeup: a state change and a notify that happen without the mutex held, racing a waiter that has checked the predicate but not yet slept. Those bugs survive a literal translation to Rust's Condvar perfectly intact.

ASIO topology. How many io_contexts, how many threads call run() on each, which handlers are wrapped in a strand and which are not. The unstranded handlers that touch shared state are your race conditions. Also list every shared_from_this() — that's C++'s answer to "the callback might outlive the object", and each one becomes a decision about ownership in Rust.

Lock-free structures. List them, list the memory orderings used, and find out why each one exists. In my experience about half exist because someone measured contention, and the other half exist because someone read a paper. The second half become channels.

Blocking calls hiding in async code. sleep_for inside a handler, synchronous file I/O, a blocking database client, a std::future::get(). In ASIO these quietly reduce throughput on one thread. In Tokio they can wedge the runtime. Flag every one.

Tooling that helps: clang-tidy with the concurrency checks enabled, and a ThreadSanitizer build of the existing C++ run against its own integration tests. You will find real bugs in the C++ before you've written any Rust. Budget time to decide, per bug, whether to fix it in C++ now or let the port fix it; don't let that decision happen by accident.

I always end up with a table shaped roughly like this (a few real-ish rows for flavour):

The most useful artifact, though, is the list of contracts the table implies: "handler H and handler G are never concurrent", "g_ring has exactly one producer", "Store is never destroyed while Flusher is running". These are the invariants C++ enforced by convention and Rust will enforce by type. Having them written down is what lets you choose Mutex<T> vs. owning task vs. channel deliberately in the sections that follow, instead of reaching for Arc<Mutex<Everything>> and hoping.

The mental-model shift: data races are gone, race conditions are not

You already know Send and Sync. The shift I want to describe is about reading the C++ with the right lens, because the C++ and the Rust are closer than they look and that closeness is a trap.

The memory model is the same. Rust's atomics are the C++20 memory model, deliberately. memory_order_acquire is Ordering::Acquire, fences are fences, and the reasoning you did about happens-before in the C++ carries over word for word. What differs is enforcement. In C++, a data race on a non-atomic is undefined behaviour and the compiler trusts you not to have one. In Rust, a data race on a non-atomic is a compile error unless you've written unsafe to get there. That's the entire safety story for concurrency, and it's a big one, but it's narrower than people assume.

Rust eliminates data races. It does not eliminate race conditions. Deadlocks compile. Lost wakeups compile. Check-then-act across two separate lock acquisitions compiles. Ordering bugs between two independent tasks compile. This one, which I ported from C++ almost verbatim, compiled and shipped:

// Two locks, two acquisitions, one race. if !registry.lock().unwrap().contains(&id) { // another task registers `id` right here registry.lock().unwrap().insert(id, conn); } 

The fix is the same as it would have been in C++ (hold the lock across the check and the insert, or use the entry API). The borrow checker had no opinion.

C++ mutexes protect code; Rust mutexes protect data. This is the single difference that reshapes architecture. A std::mutex mu_; next to three fields and a comment says nothing about which fields it covers. Mutex<T> is the field. Every time you port a struct, you are forced to answer "which lock protects this?" for every member, and the audit's contract list is where those answers come from. A C++ class that looks like this:

class Session { std::mutex mu_; std::deque<Msg> outbox_; // protected by mu_ std::atomic<bool> closed_{false}; Stats stats_; // "only touched from the io thread" (a comment) }; 

becomes something like this, and the comment becomes a type error the first time a second task touches stats:

struct Session { outbox: Mutex<VecDeque<Msg>>, closed: AtomicBool, stats: Stats, // Cell inside → !Sync → Arc<Session> won't cross threads. Good. } 

That error is the compiler doing the audit for you. Don't paper over it with a Mutex reflexively; sometimes the right answer is that stats belongs to the one task that owns the session, and the struct should be split.

&mut is a thing C++ cannot say. There is no way in C++ to express "I am the only one touching this right now"; it's all convention. In Rust every &mut self method is that claim, checked. A lot of C++ methods that were "safe to call from any thread, as long as you hold mu_" become methods on the guarded type that only exist once you've locked.

Thread lifetime is in the signature. std::thread::spawn requires 'static, so the detached-thread-references-a-stack-object bug from the audit won't compile. std::thread::scope is the tool for threads that must end before the function returns, and it's the right translation for most "spawn N workers, join them at the end" C++ blocks.

What the compiler still can't see. It's worth being explicit, because this list is where your bugs will be:

• lock ordering between unrelated Mutexes
• whether a Relaxed load is enough (memory ordering is still your job)
• every invariant inside an unsafe block, which is where your lock-free code is going to live
• cancellation, which is new: a C++ thread runs to completion unless you tell it otherwise, a Rust future can be dropped between any two .awaits

The practical rule I've landed on: translate the invariant, not the code. The C++ says mutex m; int a; int b;. The Rust has to say whether a and b are under the same lock, under different locks, or owned by different tasks. If you can't answer from the audit, that's a question for the original author, and it's worth a meeting.

Porting std::thread, mutex and condition_variable

This is the easiest of the three styles to port and the one where a literal translation does the most damage, because it looks like it worked.

Most hand-rolled thread pools are a channel

Every C++ codebase I've inherited had at least one of these:

class Pool { std::vector<std::thread> workers_; std::deque<std::function<void()>> q_; std::mutex mu_; std::condition_variable cv_; bool stop_ = false; public: explicit Pool(size_t n) { for (size_t i = 0; i < n; ++i) workers_.emplace_back([this] { for (;;) { std::function<void()> job; { std::unique_lock lk(mu_); cv_.wait(lk, [&] { return stop_ || !q_.empty(); }); if (stop_ && q_.empty()) return; job = std::move(q_.front()); q_.pop_front(); } job(); } }); } void submit(std::function<void()> f) { { std::lock_guard lk(mu_); q_.push_back(std::move(f)); } cv_.notify_one(); } ~Pool() { { std::lock_guard lk(mu_); stop_ = true; } cv_.notify_all(); for (auto& t : workers_) t.join(); } }; 

The mutex, the condvar and the stop flag are all there to implement one thing: a blocking MPMC queue that can be closed. Rust has that as a library type. The port I actually shipped:

use crossbeam_channel::{unbounded, Sender}; use std::thread::{self, JoinHandle}; type Job = Box<dyn FnOnce() + Send + 'static>; struct Pool { tx: Option<Sender<Job>>, workers: Vec<JoinHandle<()>>, } impl Pool { fn new(n: usize) -> Self { let (tx, rx) = unbounded::<Job>(); let workers = (0..n) .map(|_| { let rx = rx.clone(); // `rx.iter()` ends when every Sender is dropped. That's the stop flag. thread::spawn(move || for job in rx.iter() { job() }) }) .collect(); Pool { tx: Some(tx), workers } } fn submit(&self, f: impl FnOnce() + Send + 'static) { self.tx.as_ref().expect("pool closed").send(Box::new(f)).ok(); } } impl Drop for Pool { fn drop(&mut self) { drop(self.tx.take()); // closes the channel; workers drain and exit for w in self.workers.drain(..) { let _ = w.join(); } } } 

The condvar is gone, the stop flag is gone, and so is a bug the C++ version had: one of the pools could be destroyed from a worker thread (a job that dropped the last shared_ptr<Pool>), which made the destructor join() itself and deadlock. In the Rust version that's still possible, but JoinHandle::join from the same thread panics (the underlying pthread\_join\ reports a deadlock) instead of hanging silently, and the 'static bound on jobs makes it much harder to arrange by accident.

If the C++ pool is only ever used as "spawn N workers, do the work, join", don't even write a pool — std::thread::scope plus a channel is twelve lines and no 'static gymnastics.

The mutex family

• std::mutex → std::sync::Mutex<T>. Modern std Mutex is futex-based on Linux, so the old "use parking_lot, it's faster" advice is mostly obsolete. The remaining reasons to pick parking_lot are API: no poisoning, const fn new, and an optional deadlock_detection feature that's worth turning on in debug builds while you're porting.
• Poisoning is new. If a thread panics while holding a std Mutex, the lock is poisoned and lock() returns Err. C++ has no concept here (the mutex is simply unlocked by the lock_guard destructor during unwinding). Decide once, codebase-wide: either parking_lot, or .lock().unwrap_or_else(PoisonError::into_inner) everywhere, or treat poison as fatal. Mixing policies is how you get a service that half-works after a panic.
• std::shared_mutex → RwLock<T>. Measure before keeping it. In two of three cases where the C++ used a shared_mutex, the critical sections were short enough that a plain Mutex was faster in Rust and simpler to reason about.
• std::recursive_mutex → nothing. parking_lot::ReentrantMutex exists but only gives you &T, which tells you what the real fix is: the re-entrant call path usually exists because a callback re-enters the object that invoked it. Restructure so the inner function takes the guard (or the data) as a parameter rather than re-locking.
• std::call_once / function-local statics → OnceLock / LazyLock. These are the easy ones.
• thread_local → thread_local!. The difference is you access it through .with(|v| ...) and can't take a plain reference out, which is annoying for about a day and then prevents the bug it's designed to prevent.

Condition variables

The mapping is mechanical but the predicate inverts, which has bitten me more than once:

std::unique_lock lk(mu); cv.wait(lk, [&] { return ready; }); // wait *until* ready 

let guard = cv.wait_while(mu.lock().unwrap(), |ready| !*ready).unwrap(); // wait *while* not ready 

Rust's Condvar::wait takes the guard and gives it back, so you can't accidentally wait on a condvar without the mutex. You can still write a lost wakeup: if the notifier changes the state without holding the mutex and then calls notify_one, a waiter that has checked the predicate but not yet parked misses it. Keep the rule from the audit: state change under the lock, notify after. Rust doesn't check this.

One more thing to carry over from the audit: lock ordering is still your problem. Two Mutex<T>s in a struct say nothing about which to take first. I keep the lock-order graph from the audit as a comment on the struct, and I make any function that takes two locks take them in one place, in one order, through one helper. Boring, effective.

Porting Boost.ASIO, completion handlers and C++20 coroutines to Tokio

This is where most of the calendar goes. ASIO and Tokio solve the same problem and share almost no semantics, and the places where they differ are exactly the places that don't show up until production.

The vocabulary map, and where it lies

Strands become tasks

The canonical ASIO session, which I've seen in every codebase with only the variable names changed:

class Session : public std::enable_shared_from_this<Session> { tcp::socket sock_; asio::strand<asio::any_io_executor> strand_; std::array<char, 4096> buf_; public: void start() { do_read(); } void do_read() { auto self = shared_from_this(); // keep *this alive until the handler runs sock_.async_read_some(asio::buffer(buf_), asio::bind_executor(strand_, [this, self](std::error_code ec, size_t n) { if (ec) return; // includes operation_aborted on shutdown handle(buf_.data(), n); do_read(); })); } }; 

Three things are doing work here: the strand (so handle never runs concurrently with itself), the self capture (so buf_ outlives the read), and the ec check (so cancellation is observed). The Tokio version:

async fn session(mut sock: TcpStream, shutdown: CancellationToken) { let mut buf = [0u8; 4096]; loop { tokio::select! { _ = shutdown.cancelled() => break, res = sock.read(&mut buf) => match res { Ok(0) | Err(_) => break, Ok(n) => handle(&buf[..n]), }, } } // `sock` and `buf` drop here. No self-capture, no strand, no operation_aborted branch. } 

The strand disappeared because the task is the strand: one task, one socket, sequential by construction. The self capture disappeared because sock.read(&mut buf) borrows buf for exactly the duration of the operation, and the compiler checks it. The ASIO rule "the buffer must outlive the async operation" was enforced by nothing in C++ and by the signature in Rust.

The decision you do have to make: what about state that several sessions share? In ASIO that was "a strand around the shared object". In Tokio you have three choices, and I'd pick them in this order: (1) give the state to one task and talk to it over an mpsc channel (the actor pattern — this is the honest translation of a strand), (2) Arc<Mutex<T>> with short critical sections and no .await inside, (3) tokio::sync::Mutex only if you must hold the lock across an .await, which you should be suspicious of.

C++20 coroutines look closest and are the most deceptive

asio::awaitable code reads almost like Rust:

asio::awaitable<void> session(tcp::socket sock) { std::array<char, 4096> buf; for (;;) { auto n = co_await sock.async_read_some(asio::buffer(buf), asio::use_awaitable); handle(buf.data(), n); } } 

The differences that matter when porting:

• Eager vs lazy. async_read_some starts the read when called; the co_await just waits. In Rust nothing happens until .await. If you port a pattern that starts several operations and then awaits them in turn, you'll need join!/JoinSet to get the concurrency back, or you'll silently serialise them.
• Heap vs state machine. A C++ coroutine frame is heap-allocated by default (the compiler may elide it, but rarely across a translation unit boundary). A Rust async fn is a state machine with a size known at compile time, stored wherever you put it, which is why recursion needs Box::pin and why a deeply nested async call tree can make a 40 KB future. Neither is better; they just fail differently.
• Cancellation — the next section, because it deserves one.

Cancellation is the thing that will bite you

In ASIO, cancelling an operation delivers operation_aborted to its handler, and the handler runs. Your cleanup is in the handler. In Tokio, cancelling a future means dropping it, and nothing in its body runs again. Your cleanup is in Drop, or it doesn't happen.

This sounds like a bookkeeping difference. It is the single largest source of bugs I've introduced while porting ASIO code. The failure mode is cancellation safety: a future dropped between two .awaits may have done half its work.

// NOT cancellation-safe. If `shutdown` wins the select after 3 of 8 bytes // arrived, those 3 bytes are gone and the stream is desynchronised. tokio::select! { _ = shutdown.cancelled() => {} r = sock.read_exact(&mut header) => { ... } } 

Tokio's docs mark each method as cancellation-safe or not; read them for every method you use inside a select!. The patterns that kept me out of trouble:

1. Cancel at task granularity, not operation granularity. Instead of select! around individual I/O calls, let the task run and abort() the whole thing (or signal it between messages) when you need to stop. One place to reason about, not fifty.
2. Make state transitions atomic with respect to .await. Do all your bookkeeping, then await, or await into a local and then commit. Never mutate shared state, await, and then mutate again as if nothing could have happened in between.
3. Use Drop as the cleanup hook, and keep it cheap. The ASIO operation_aborted branch that returned a buffer to a pool becomes an impl Drop on a guard type. If the cleanup needs to be async, you need a different design (a shutdown message to the owning task), because Drop can't await.
4. CancellationToken from tokio-util for structured shutdown, and JoinSet so the parent actually knows when its children are gone. Detached tasks are the Rust equivalent of detached threads.

Blocking the executor

ASIO is tolerant of a handler that blocks for five milliseconds: that one thread is busy, the others keep going. Tokio is less tolerant, and the trap when porting is more specific:

// Fine in ASIO: the handler runs later, after the lock_guard is gone. std::lock_guard lk(mu_); asio::async_write(sock_, asio::buffer(outbox_.front()), handler); 

// Not fine: the guard lives across the await. let g = self.outbox.lock().unwrap(); self.sock.write_all(&g.front().unwrap()).await; // compile error on a multi-thread runtime, deadlock on current_thread 

A std::sync::MutexGuard is !Send, so this is a compile error if the future needs to be Send (i.e. if it's tokio::spawned on the multi-thread runtime). On a current_thread runtime, or inside block_on, it compiles — and then deadlocks the moment two tasks contend. The fix is to copy the data out, drop the guard, then await. Do that in every port, unconditionally. For real blocking work (synchronous file I/O, a blocking client library), spawn_blocking.

Runtime topology is a decision, not a default

If the C++ used one io_context with N threads and strands, the multi-thread Tokio runtime is the natural mapping and work-stealing will probably improve tail latency. If the C++ used one io_context per core with SO_REUSEPORT and no cross-thread posting — common in anything that was performance-tuned — then porting to a single work-stealing runtime can be a regression, because you've just introduced cross-core traffic the original avoided. The faithful port is N current_thread runtimes, one per core, with sharded state. (And if the C++ was on io_uring, that's the moment to look at the thread-per-core runtimes built around it, rather than forcing the model onto Tokio.)

For what it's worth, I got this wrong once in each direction.

Porting lock-free atomics and hand-rolled queues

The good news first: the memory orderings map one to one, and the happens-before reasoning you already did is still valid. The bad news is that Rust'...