Building a Low-Latency Webhook Ingress Gateway with Rust and `io_uring`

Webhooks carry the events that drive modern integrations: payment notifications, CI/CD triggers, chat platform callbacks and device telemetry. An ingress gateway sits in front of all of them. It must accept bursts of small HTTP requests, authenticate them, and hand them off to downstream queues without becoming the bottleneck.

At high request rates, the cost of crossing the operating system boundary becomes a significant part of the per-request budget. This guide looks at how Linux io_uring and Rust can reduce that overhead, what zero-copy receive actually requires, and how to build a gateway skeleton around them. It also covers where the technique helps and where it doesn’t.

1. Where Traditional Webhook Ingress Spends Its Time

A conventional readiness-based server, such as one built on epoll, handles an incoming request roughly like this:

1. DMA and interrupt handling: The NIC writes the packet into kernel-owned receive buffers via DMA and signals the CPU. The kernel’s network stack processes it, typically via softirq.
2. TCP processing: The kernel validates the segment, runs it through the TCP state machine, and queues the payload on the socket’s receive queue.
3. Readiness notification: A thread blocked in epoll_wait() is woken because the socket is readable.
4. Copy to user space: The application calls read() or recv(), and the kernel copies the payload from the socket buffer into a user-space buffer.

Each of those syscalls crosses the user/kernel boundary, and each payload is copied once on the way in. CPU vulnerability mitigations introduced after Spectre and Meltdown can make these transitions more expensive on some hardware. The exact cost varies widely by CPU, kernel version and mitigation settings, so measure it on your own machines instead of relying on a single figure.

For a gateway handling very large numbers of small requests, per-request syscall overhead and per-request copies add up. Business logic is often not the main cost.

2. What io_uring Changes

io_uring was introduced in Linux 5.1. Instead of one syscall per operation, the application and kernel share two ring buffers in memory:

• Submission Queue (SQ): the application writes Submission Queue Entries (SQEs) describing operations such as accept, recv or send.
• Completion Queue (CQ): the kernel writes Completion Queue Entries (CQEs) as operations finish.

The application can queue many operations and submit them with a single io_uring_enter call. That call can also wait for completions, so syscall counts drop sharply compared with a readiness-plus-read loop.

Application (user space) writes SQEs ──► [ Submission Queue ] ──┐ │ shared memory rings reads CQEs ◄── [ Completion Queue ] ◄─┘ │ Kernel: io_uring subsystem executes the operations 

Features That Matter for Ingress

• Multishot receive: Since Linux 6.0, a single receive request can keep producing completions as data arrives, so you don’t re-submit an SQE for every read. Multishot receive uses provided buffers: you register a buffer group, and the kernel picks a buffer from it for each completion.
• IORING_SETUP_SINGLE_ISSUER: declares that only one thread submits requests to the ring, which lets the kernel skip some synchronization.
• IORING_SETUP_DEFER_TASKRUN: defers completion work until the application explicitly enters the kernel to wait for events. The liburing documentation recommends it as the default for new applications that run one ring per thread, and it requires SINGLE_ISSUER.
• Zero-copy receive (IORING_OP_RECV_ZC): merged in Linux 6.15, covered in detail below.

3. Zero-Copy Receive: What It Is and What It Requires

io_uring zero-copy receive (ZC Rx) removes the kernel-to-user copy on the receive path, so packet payloads arrive directly in user-space memory. Packet headers are still processed by the kernel TCP stack as normal, so unlike kernel-bypass frameworks such as DPDK you keep the kernel’s TCP implementation.

It is not a drop-in switch. Per the kernel documentation, it depends on all of the following:

• Linux 6.15 or newer. The feature was merged for that release.
• NIC hardware features. The NIC must support header/data split, so headers land in kernel memory and payloads land in user memory. It must also support flow steering and RSS, so that only the flows you choose reach the hardware queue dedicated to zero-copy, and all other traffic is steered away from it. At the time of the initial merge the kernel did not configure the NIC for you, so you do this out of band. Check your NIC and driver documentation before planning around it.
• Specific ring setup flags. The ring must be created with IORING_SETUP_SINGLE_ISSUER, IORING_SETUP_DEFER_TASKRUN, and large completion entries (IORING_SETUP_CQE32, or IORING_SETUP_CQE_MIXED in newer kernels).
• Registered memory and a refill ring. You register a user-space memory area and bind it to a specific hardware receive queue. After processing, you return buffers to the kernel through a shared refill queue. Buffer lifetimes are managed by io_uring, with different rules from ordinary network buffers.

An example NIC setup following the kernel documentation looks like this:

# Make sure there are at least two hardware queues sudo ethtool -L eth0 combined 2 # Enable header/data split sudo ethtool -G eth0 tcp-data-split on # Reserve half of the queues for zero-copy using RSS sudo ethtool -X eth0 equal 1 # Steer the flows you want into the zero-copy queue (queues are 0-indexed) sudo ethtool -N eth0 flow-type tcp4 dst-port 8443 action 1 

Adapt the interface name, queue counts and flow rules to your hardware.

A Realistic Expectation

Zero-copy receive saves a memory copy per payload. Webhook payloads are typically small, often a few hundred bytes to a few kilobytes, so the copy itself is usually a small part of the cost. For small requests, most of the gain from io_uring comes from fewer syscalls, batching and multishot operations, not from zero-copy. Zero-copy receive matters more as payload sizes and bandwidth grow. Start with the portable features and add ZC Rx only if profiling shows the copy is significant and your hardware supports it.

4. Designing the Rust Ingress Engine

The io-uring crate from the tokio-rs organization is a low-level Rust binding to the io_uring interface. Higher-level crates such as tokio-uring build a Tokio-compatible runtime on top of it. tokio-uring runs a current-thread runtime, and its documentation recommends spawning multiple threads, each with its own runtime, for concurrency. That matches the architecture below.

Architectural Blueprint

1. Thread-per-core. Run one worker per core, each with its own ring, and pin workers with a crate such as core_affinity. Each thread then keeps its data hot in its own caches, and SINGLE_ISSUER holds naturally because only one thread touches each ring.
2. Listener distribution. Give each worker its own listening socket bound with SO_REUSEPORT, so the kernel spreads incoming connections across workers without a shared accept queue.
3. Provided buffers and multishot receive. Register a buffer group per ring and use multishot receive so one SQE keeps producing completions.
4. Batch submissions. Queue several SQEs and submit them together instead of calling into the kernel per operation.
5. Bounded work on the hot path. Do only parsing, authentication and enqueueing on the I/O thread, and push anything slower to other threads.

5. Implementation Sketch

The snippets below show the shape of the core of the engine. They are illustrative and not production-ready. Pin your crate versions, check the current io-uring API on docs.rs, and run the opcode probe at startup to confirm your kernel supports what you use.

Cargo Dependencies

[dependencies] io-uring = "0.7" libc = "0.2" core_affinity = "0.8" ring = "0.17" 

Creating the Ring

use io_uring::IoUring; use std::io; pub fn create_ingress_ring(entries: u32) -> io::Result<IoUring> { // One ring per worker thread. SINGLE_ISSUER declares that only this // thread submits; DEFER_TASKRUN requires it and defers completion work // until this thread enters the kernel to wait for events. IoUring::builder() .setup_single_issuer() .setup_defer_taskrun() .build(entries) } 

Submitting a Multishot Receive

Multishot receive selects its buffer from a registered buffer group. You must provide that group to the kernel first, either with ProvideBuffers or with a ring-mapped buffer group registered via the buffer ring API.

use io_uring::{opcode, types, IoUring}; use std::io; use std::os::unix::io::RawFd; pub fn submit_multishot_recv( ring: &mut IoUring, fd: RawFd, buf_group: u16, user_data: u64, ) -> io::Result<()> { // One SQE; the kernel keeps posting CQEs as data arrives, picking a // buffer from `buf_group` each time. Multishot recv needs Linux 6.0+. let sqe = opcode::RecvMulti::new(types::Fd(fd), buf_group) .build() .user_data(user_data); // Safety: the fd and buffer group must stay valid while the operation is in flight. unsafe { ring.submission() .push(&sqe) .map_err(|_| io::Error::new(io::ErrorKind::Other, "submission queue full"))?; } Ok(()) } 

When a multishot request terminates, the final CQE does not have the IORING_CQE_F_MORE flag set. Check for it and re-arm the receive when it disappears. This happens, for example, when the buffer group runs out of buffers. Return buffers to the group once you have finished with them.

To go further and use zero-copy receive, you register the interface queue, the memory area and the refill ring described in section 3. The kernel documentation shows this using the C liburing API. Check the current state of Rust bindings before committing to it, and consider calling into liburing or using raw registration if the crate doesn’t cover it yet.

6. Security and Backpressure

A webhook gateway must authenticate payloads and protect itself under load.

Signature Verification

Many providers sign the raw request body with HMAC-SHA256. GitHub, for example, sends the signature in the X-Hub-Signature-256 header as sha256= followed by a hex-encoded digest. Always compute the HMAC over the exact raw bytes received, before any JSON parsing or re-serialization, and compare in constant time.

The ring crate’s hmac::verify performs the comparison in constant time for you:

use ring::hmac; /// `payload` is the raw request body; `sig_hex` is the header value /// with the `sha256=` prefix already stripped. pub fn verify_webhook_signature(key: &hmac::Key, payload: &[u8], sig_hex: &[u8]) -> bool { // Decode hex into a fixed-size stack buffer (SHA-256 = 32 bytes). let mut tag = [0u8; 32]; if sig_hex.len() != 64 || hex_decode(sig_hex, &mut tag).is_err() { return false; } hmac::verify(key, payload, &tag).is_ok() } fn hex_decode(src: &[u8], dst: &mut [u8; 32]) -> Result<(), ()> { fn nibble(b: u8) -> Result<u8, ()> { match b { b'0'..=b'9' => Ok(b - b'0'), b'a'..=b'f' => Ok(b - b'a' + 10), b'A'..=b'F' => Ok(b - b'A' + 10), _ => Err(()), } } for (i, pair) in src.chunks_exact(2).enumerate() { dst[i] = (nibble(pair[0])? << 4) | nibble(pair[1])?; } Ok(()) } 

Verifying on a stack-allocated tag avoids heap allocation for the comparison. Reject signatures of the wrong length early, and consider also checking a timestamp or delivery ID where the provider supplies one, to limit replay attacks.

Backpressure and Load Shedding

When downstream queues fill up, the gateway should shed load deliberately instead of buffering until it runs out of memory. The thresholds below are starting points to tune, not universal values.

Sensible webhook senders retry on 5xx responses, so fast rejection is usually safer than accepting data you can’t process. If the CQ overflows, the kernel keeps excess completions in an internal backlog on modern kernels, which costs memory and latency. Size the CQ generously and drain it promptly.

7. Measuring Instead of Assuming

Published numbers for io_uring versus epoll vary widely depending on workload, payload size, kernel version, NIC and mitigations, so benchmark your own gateway on your own hardware. A few practical points:

• Measure tail latency, not averages. Record p50, p99 and p99.9 with a load generator that avoids coordinated omission, such as wrk2, and with realistic payload sizes and connection counts.
• Compare like with like. Test an epoll-based Tokio server and your io_uring server on the same machine, same kernel, same NIC settings and same business logic.
• Run controlled experiments. Toggle one feature at a time (multishot, DEFER_TASKRUN, core pinning, ZC Rx) so you can attribute gains.
• Watch the whole system. Check CPU utilization per core, softirq time, context switches and syscall counts using tools such as perf, bpftrace and ethtool -S.
• Define your latency target precisely. State where the clock starts and stops, for example from request arrival at the NIC to enqueue downstream, because “sub-millisecond” is meaningless without that boundary.

8. Operational Considerations

• Kernel and platform support. io_uring is Linux-only, and some container runtimes, seccomp profiles and managed platforms restrict or disable it because of its security history. Verify that your target environment permits it. Use the opcode probe at startup to detect unsupported operations and fall back gracefully.
• Kernel versions. Multishot receive needs 6.0 or newer, and zero-copy receive needs 6.15 or newer plus NIC support. Plan a fallback path (plain multishot receive without ZC) for nodes that lack either.
• Observability. Export per-worker metrics such as SQ and CQ depth, buffer-group exhaustion events, re-arm counts, 503 rates and downstream queue depth.
• TLS. Most real webhook traffic is HTTPS. Terminating TLS at the gateway adds substantial CPU cost and changes the picture, because decrypted data must exist in user space anyway. Many deployments terminate TLS at a load balancer or proxy in front of the gateway. That also affects how much zero-copy receive can help.
• Complexity. A custom io_uring engine takes on responsibility for HTTP parsing, buffer lifecycle and safety around unsafe code. Evaluate whether a mature Tokio-based server meets your latency goals before building one.

9. Conclusion

io_uring gives a Rust ingress gateway real tools for reducing kernel-boundary overhead: batched submissions, multishot receive, and deferred completion processing on single-issuer rings. Thread-per-core design and careful buffer management build on those. Zero-copy receive, available from Linux 6.15, can additionally remove the receive-side copy. It also demands specific NIC features, out-of-band queue and flow configuration, and a different buffer lifecycle.

Treat each of these as an optimization you earn through measurement. Start with a pinned thread-per-core design using multishot receive, benchmark it against a plain Tokio baseline on your own hardware, and adopt zero-copy receive only if your payload sizes and profiling justify it. Backpressure, constant-time signature verification and a clearly defined latency target will do as much for a production gateway as any individual kernel feature.

Originally published at https://instawebhook.com/blog/building-a-low-latency-webhook-ingress-gateway-with-rust-and-io-uring