Building a fast, resilient tunnel client means working around limits at three distinct layers:

  1. Kernel-to-userspace overhead: every byte a proxy forwards is copied across the kernel/user boundary twice, and each chunk costs two system calls.
  2. Network topology: hard (endpoint-dependent) NATs and Carrier-Grade NAT (CGNAT) get in the way of direct peer-to-peer paths.
  3. Transport latency: a fresh TCP + TLS connection spends round trips on handshakes before the first request byte can be sent, which hurts short-lived requests.

This guide covers the kernel primitives, protocols and design trade-offs for each layer. It also says where each technique stops helping, because the three layers interact in ways that are easy to get wrong. Every code sample below was compiled and run, either against a loopback echo test or against a mock server.

Part 1: Cutting the user-space copy with splice()

A conventional user-space forwarder reads from one socket into an application buffer, then writes that buffer to another socket:

[ Traditional forwarding ] Kernel : [ Socket A recv buffer ] --read()--> (copy) --+ v User : [ Application buffer ] | Kernel : [ Socket B send buffer ] <--write()-- (copy) -+

For bulk transfers this costs two copies per chunk, two system calls per chunk, and a user-space buffer that competes for CPU cache with the rest of your program.

What splice() does

Per the Linux man page, splice() moves data between two file descriptors without copying between kernel address space and user address space. At least one of them must be a pipe. The call has been in Linux since 2.6.17, and since 2.6.31 both descriptors may be pipes. Together with vmsplice(2) and tee(2), it gives a program control over a kernel buffer that is implemented with the same buffer type used for pipes.

[ Splice pipeline ] Kernel : [ Socket A ] --splice--> [ Pipe buffer ] --splice--> [ Socket B ] User : (no payload bytes ever visible here)

The signature:

#define _GNU_SOURCE #include <fcntl.h>  ssize_t splice(int fd_in, loff_t *off_in, int fd_out, loff_t *off_out, size_t len, unsigned int flags);

Three details are commonly misstated:

  • SPLICE_F_MOVE does nothing on modern kernels. The man page says the original implementation was buggy, so since Linux 2.6.21 the flag is accepted but is a no-op. Don’t rely on it for page-moving behaviour.
  • SPLICE_F_NONBLOCK makes the pipe side of the operation non-blocking. The call may still block if the other descriptor is a blocking socket, so set O_NONBLOCK on the sockets too.
  • SPLICE_F_MORE hints that more data follows in a later splice(), which is useful on the pipe-to-socket leg.

“No copy to user space” is not the same as “zero copies anywhere”. The man page promises only the former.

Where vmsplice() fits (and where it doesn’t)

vmsplice(2) connects user memory to a pipe. When the pipe is open for writing, it maps your memory ranges into the pipe. When the pipe is open for reading, it actually copies data out into your buffers. That makes vmsplice relevant when your own code produces data (for example, a pre-built frame), not for socket-to-socket forwarding, where nothing needs to touch user space at all.

The caveat is that because the pipe maps your pages rather than copying them, you must not modify that memory until the kernel has consumed it. SPLICE_F_GIFT goes further and declares the pages handed to the kernel for good, so the application must treat them as read-only from then on. It also requires page-aligned addresses and lengths.

When splice helps, and when it can’t

Splice only applies when the tunnel forwards bytes it does not need to read or transform:

  • Plain TCP-to-TCP forwarding (a TCP tunnel carrying an opaque stream, or a TLS-passthrough path): splice fits.
  • TLS terminated in user space, or QUIC: the application must see plaintext to encrypt or decrypt it, so splice cannot help. The kernel’s TLS offload (kTLS) can narrow this gap for TCP: its documentation describes sendfile support on the transmit path, so plaintext can reach a TLS socket without a user-space copy.
  • QUIC and other UDP protocols can’t use splice, so the available levers are batching. Google’s QUICHE library, for example, uses UDP GSO (UDP_SEGMENT) so one sendmsg can carry multiple segments, and sendmmsg batching to send several packets in one call (QUICHE source).
  • io_uring also exposes splice through io_uring_prep_splice(3), which is worth knowing if you are already building on a completion-based loop.

Go: use the standard library first, hand-roll only to tune

Go’s net package already splices on Linux. When a *net.TCPConn is the source of an io.Copy into another TCP connection, the runtime takes a splice path internally and falls back to a regular copy when it does not apply (net/splice_linux.go). For most proxies, io.Copy(dst, src) between two *net.TCPConn values is the right answer.

A hand-written version is only worth it when you need control the stdlib doesn’t give you, such as pipe sizing, chunk size or per-connection accounting. The version below integrates with Go’s netpoller through syscall.RawConn, so a would-block parks the goroutine instead of spinning:

//go:build linux package main import ( "errors" "fmt" "net" "golang.org/x/sys/unix" ) // SplicePipe is a kernel pipe used as the staging buffer between two sockets. type SplicePipe struct{ r, w int } // NewSplicePipe creates a non-blocking pipe and resizes it. Unprivileged // processes are capped by /proc/sys/fs/pipe-max-size (1 MiB by default). func NewSplicePipe(size int) (*SplicePipe, error) { var fds [2]int if err := unix.Pipe2(fds[:], unix.O_CLOEXEC|unix.O_NONBLOCK); err != nil { return nil, fmt.Errorf("pipe2: %w", err) } if _, err := unix.FcntlInt(uintptr(fds[0]), unix.F_SETPIPE_SZ, size); err != nil { unix.Close(fds[0]) unix.Close(fds[1]) return nil, fmt.Errorf("F_SETPIPE_SZ(%d): %w", size, err) } return &SplicePipe{r: fds[0], w: fds[1]}, nil } func (p *SplicePipe) Close() { unix.Close(p.r); unix.Close(p.w) } // Copy moves bytes from src to dst until src reaches EOF. func (p *SplicePipe) Copy(dst, src *net.TCPConn, chunk int) (int64, error) { srcRC, err := src.SyscallConn() if err != nil { return 0, err } dstRC, err := dst.SyscallConn() if err != nil { return 0, err } var total int64 for { // Stage 1: socket -> pipe. var n int64 var serr error rerr := srcRC.Read(func(fd uintptr) bool { for { n, serr = unix.Splice(int(fd), nil, p.w, nil, chunk, unix.SPLICE_F_NONBLOCK) if serr != unix.EINTR { break } } // false => runtime waits until the socket is readable, then retries return !errors.Is(serr, unix.EAGAIN) }) if rerr != nil { return total, rerr } if serr != nil { return total, fmt.Errorf("splice socket->pipe: %w", serr) } if n == 0 { return total, nil // EOF } // Stage 2: pipe -> socket. Drain everything just staged. for left := n; left > 0; { var w int64 werr := dstRC.Write(func(fd uintptr) bool { for { w, serr = unix.Splice(p.r, nil, int(fd), nil, int(left), unix.SPLICE_F_NONBLOCK|unix.SPLICE_F_MORE) if serr != unix.EINTR { break } } return !errors.Is(serr, unix.EAGAIN) }) if werr != nil { return total, werr } if serr != nil { return total, fmt.Errorf("splice pipe->socket: %w", serr) } left -= w total += w } } }

Use one pipe per direction per connection. When Copy returns, call dst.CloseWrite() so the EOF propagates as a half-close. A 64 MiB random payload sent through a two-hop loopback echo using this code came back byte-for-byte identical.

Rust: splice inside a Tokio event loop

In Rust, the nix crate wraps splice, but its splice and SpliceFFlags are gated behind the zerocopy Cargo feature (the code below was built against nix 0.30). Tokio’s TcpStream::async_io is the clean way to run a raw syscall under its readiness tracking: if the closure returns WouldBlock, Tokio clears readiness and waits again.

[dependencies] nix = { version = "0.30", features = ["fs", "zerocopy"] } tokio = { version = "1", features = ["rt-multi-thread", "net", "macros", "io-util"] } 
use std::io; use std::os::fd::OwnedFd; use nix::fcntl::{fcntl, splice, FcntlArg, OFlag, SpliceFFlags}; use nix::unistd::pipe2; use tokio::io::Interest; use tokio::net::TcpStream; /// A kernel pipe used as the staging buffer between two sockets. pub struct SplicePipe { r: OwnedFd, w: OwnedFd, } impl SplicePipe { pub fn new(size: i32) -> io::Result<Self> { let (r, w) = pipe2(OFlag::O_CLOEXEC | OFlag::O_NONBLOCK)?; fcntl(&w, FcntlArg::F_SETPIPE_SZ(size))?; Ok(Self { r, w }) } /// Move bytes from `src` to `dst` until `src` hits EOF. pub async fn forward(&self, src: &TcpStream, dst: &TcpStream, chunk: usize) -> io::Result<u64> { let mut total = 0u64; loop { // socket -> pipe (EAGAIN converts to WouldBlock, so Tokio re-waits) let n = src .async_io(Interest::READABLE, || { splice(src, None, &self.w, None, chunk, SpliceFFlags::SPLICE_F_NONBLOCK) .map_err(io::Error::from) }) .await?; if n == 0 { return Ok(total); // EOF } // pipe -> socket: drain everything just staged let mut left = n; while left > 0 { let w = dst .async_io(Interest::WRITABLE, || { splice( &self.r, None, dst, None, left, SpliceFFlags::SPLICE_F_NONBLOCK | SpliceFFlags::SPLICE_F_MORE, ) .map_err(io::Error::from) }) .await?; left -= w; total += w as u64; } } } }

Wrap each TcpStream in an Arc so one task can forward in each direction, and shut down the destination’s write half when a direction finishes. The same 64 MiB loopback echo test passed with this code.

Practical guidance

  • Measure on your workload. Splice removes the user-space copy and the per-chunk syscall pair, but it adds pipe management and (in this design) a second syscall per chunk, so small messages or heavily transformed streams may not benefit.
  • The default pipe capacity is 64 KiB. Raising it with F_SETPIPE_SZ is a throughput knob, bounded by /proc/sys/fs/pipe-max-size for unprivileged processes.
  • Linux-only: splice is Linux-specific, so keep a portable fallback (io.Copy, tokio::io::copy) for other platforms.

Part 2: Traversing NAT and CGNAT

Why carrier-grade NAT is different

Most home routers perform one layer of NAT. Under CGNAT, the ISP adds a second layer. RFC 6598 (BCP 153) reserved 100.64.0.0/10 as Shared Address Space for service providers to number the links between their CGN devices and customer equipment. If your router’s WAN address falls in that range, you are almost certainly behind CGNAT.

[ CGNAT: two NATs in series ] Laptop (192.168.1.50) └─> Home router NAT WAN address: 100.64.10.15 (RFC 6598 shared space) └─> ISP CGNAT public address: 198.51.100.1:45210 (documentation address) └─> Internet

(The 198.51.100.x and 203.0.113.x addresses in this article are reserved documentation addresses, not real hosts.)

NAT behaviour: two independent properties

The older “full cone / restricted cone / port-restricted cone / symmetric” vocabulary dates from early STUN-era work. Modern explanations, such as Tailscale’s, describe NAT behaviour with two properties that matter for traversal, and treat the named categories as a rough map onto them:

  • Mapping: does the external port depend on the destination? If it doesn’t, the NAT is “easy”; if it does, it’s “hard”.
  • Inbound filtering: which remote addresses may send packets back through a mapping?
Legacy name Mapping Inbound filtering Direct UDP traversal
Full cone Same external port for every destination Any remote host Easy
Address-restricted cone Same external port for every destination Only IPs the host has already sent to Easy once each side has sent first
Port-restricted cone Same external port for every destination Only IP:port pairs the host has already sent to Easy once each side has sent first
Symmetric New external port per destination Only the exact IP:port contacted Hard

For direct paths, you need at least an easy-mapping NAT on one side. Home routers tend to be easy NATs, while hard NATs tend to be office routers and cloud NAT gateways.

STUN, rendezvous and hole punching

RFC 8489 (which obsoletes RFC 5389, February 2020) defines STUN as a helper tool: an endpoint can use it to learn the public address and port its NAT assigned, to check connectivity, and to keep NAT mappings alive. The RFC is explicit that STUN is a tool used inside a complete traversal solution such as ICE, not a solution by itself.

That corrects a common simplification: the STUN server does not exchange addresses between peers. A separate rendezvous (signalling) channel, which in a tunnel product is usually your control plane, carries each side’s observed public endpoint to the other.

  1. Both endpoints send STUN binding requests and learn their public ip:port (e.g. 198.51.100.1:10050 and 203.0.113.8:20080).
  2. Each reports its endpoint to the rendezvous service, which hands each peer the other’s endpoint.
  3. Both peers send UDP packets to each other’s public endpoint at about the same time. Each outbound packet creates NAT state that lets the other side’s packets in. The first packet in one direction may be dropped, which is expected.
  4. Once packets flow, keep sending keepalives so the mappings don’t expire. 
[ UDP hole punching ] Client A STUN server Rendezvous service Client B |--- binding req ---->| | | |<-- A_pub=198.51.100.1:10050 |<--- binding req -----| | | | B_pub=203.0.113.8:20080 |--- register A_pub ------------------->|<---- register B_pub -| |<-- B_pub ---------------------------|--- A_pub ----------->| | | |=== punch A -> B (may be dropped by B's NAT) ===============>| |<============= punch B -> A (A's NAT now has state) ========| |<================== direct UDP path ========================>|

Hard NATs: the birthday-paradox trick, with real numbers

When one side has a hard NAT and the other an easy NAT, the easy side doesn’t know which port to send to. Tailscale’s write-up shows how the birthday paradox helps: the hard side opens many sockets (say 256), each of which gets its own random external port, while the easy side probes random destination ports. The numbers in that post: with 256 ports open on the hard side, about 174 random probes give roughly a 50% chance of a match, 256 probes give 64%, 1,024 give 98%, and 2,048 give 99.9%.

When both sides are hard, the search space becomes pairs of ports, and the odds collapse. In the same analysis, success after 20 seconds at the same rates is around 0.01%, and reaching 99.9% needs roughly 170,000 probes per side (about 28 minutes at 100 packets per second). The author also warns that aggressive probing can exhaust the session table of small NAT devices.

The practical conclusion is that hard-NAT-to-hard-NAT paths generally need a relay.

Port-mapping protocols: useful, but only at the nearest NAT

UPnP IGD, NAT-PMP and PCP all let a host ask its gateway to forward a port. They share a request/response pattern: ask the gateway to forward a WAN port to a LAN address, and receive the assigned public address and port. The important limitation, from the same Tailscale material, is that they act only on the NAT closest to the client. In a double-NAT arrangement, they cannot reach the outer NAT, so they usually fail. Behind CGNAT, a UPnP or NAT-PMP mapping on your home router only opens a port on the router’s 100.64.x.x WAN side, which doesn’t make you reachable from the internet.

PCP (RFC 6887) is the successor to NAT-PMP and supports both IPv4 and IPv6. Its useful features for tunnel clients:

  • Servers listen on UDP port 5351; clients listen on UDP 5350 for multicast ANNOUNCE messages.
  • The MAP opcode requests an inbound mapping. The suggested external port and address are hints; the server may assign something else.
  • The PREFER_FAILURE option tells the server not to create an alternative mapping if the suggested one isn’t available. When the server can’t honour the request under that option, it returns CANNOT_PROVIDE_EXTERNAL (result code 11).
  • UNSUPP_VERSION (result code 1) means the server doesn’t speak the PCP version you used.

Whether PCP can reach a CGN depends on whether the ISP runs a PCP server for subscribers, which a client can’t know in advance. Treat PCP as an opportunistic first attempt, not a guaranteed fallback.

PCP MAP request layout (RFC 6887 sections 7.1 and 11.1)

A MAP request is 60 bytes: a 24-byte common header plus a 36-byte MAP body.

 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Version = 2 |R| Opcode=1 | Reserved | +---------------+---------------+-------------------------------+ | Requested Lifetime (32 bits) | +---------------------------------------------------------------+ | | | PCP Client's IP Address (128 bits) | | | | | +---------------------------------------------------------------+ | | | Mapping Nonce (96 bits) | | | +---------------+-----------------------------------------------+ | Protocol | Reserved (24 bits) | +-------------------------------+-------------------------------+ | Internal Port | Suggested External Port | +-------------------------------+-------------------------------+ | | | Suggested External IP Address (128 bits) | | | | | +---------------------------------------------------------------+

IPv4 addresses are carried as IPv4-mapped IPv6 (::ffff:a.b.c.d). In a response the “R” bit is set, the header gains a result code and an epoch time, and the MAP body carries the assigned external port and address in place of the suggestions.

Go: a PCP MAP client 

package main import ( "bytes" "crypto/rand" "encoding/binary" "fmt" "net" "time" ) const ( pcpServerPort = 5351 pcpVersion = 2 opMap = 1 protoUDP = 17 pcpReqLen = 60 // 24-byte common header + 36-byte MAP body ) // PCPMapping is what the PCP server granted. type PCPMapping struct { ExternalIP net.IP ExternalPort uint16 Lifetime time.Duration } // RequestMapping asks the PCP server at gateway to forward external UDP // traffic to localIP:internalPort (RFC 6887, MAP opcode). func RequestMapping(gateway, localIP net.IP, internalPort uint16, lifetime uint32) (*PCPMapping, error) { // Bind to localIP: the PCP Client IP field must match the packet's source // address, otherwise the server rejects the request. conn, err := net.DialUDP("udp", &net.UDPAddr{IP: localIP}, &net.UDPAddr{IP: gateway, Port: pcpServerPort}) if err != nil { return nil, err } defer conn.Close() var nonce [12]byte if _, err := rand.Read(nonce[:]); err != nil { return nil, err } req := make([]byte, pcpReqLen) req[0] = pcpVersion // version req[1] = opMap // R bit = 0 (request), opcode = 1 binary.BigEndian.PutUint32(req[4:8], lifetime) copy(req[8:24], localIP.To16()) // client IP (IPv4-mapped for IPv4) copy(req[24:36], nonce[:]) // mapping nonce req[36] = protoUDP // bytes 37-39 reserved binary.BigEndian.PutUint16(req[40:42], internalPort) binary.BigEndian.PutUint16(req[42:44], internalPort) // suggested external port (a hint) // bytes 44-59: suggested external IP; zeros = no preference _ = conn.SetDeadline(time.Now().Add(3 * time.Second)) if _, err := conn.Write(req); err != nil { return nil, err } resp := make([]byte, 1100) // PCP messages are at most 1100 bytes n, err := conn.Read(resp) if err != nil { return nil, fmt.Errorf("no PCP response: %w", err) } return parseMapResponse(resp[:n], nonce) } func parseMapResponse(b []byte, nonce [12Source：https://dev.to/instatunnel/high-performance-edge-tunnelling-zero-copy-io-nat-traversal-and-quic-0-rtt-architecture-43nh
My favorites
PreviousOpenAI Models Land on Google Cloud: GPT-6 on Model GardenNextThe first "actual" bug was a moth: 4 debugging steps that still work

Related posts

Google Cloud Spend Caps Reach General Availability With an API and Live Costs

Google Cloud Spend Caps Reach General Availability With an API and Live Costs

dev.to
I built a Tinder-style Website for finding coding project ideas

I built a Tinder-style Website for finding coding project ideas

dev.to
GSAP ScrollTrigger image sequence in WordPress without code: when DIY isn't worth it

GSAP ScrollTrigger image sequence in WordPress without code: when DIY isn't worth it

dev.to
Surviving AWS Spot Interruptions With Real Checkpointing

Surviving AWS Spot Interruptions With Real Checkpointing

dev.to
Forward Watch

Forward Watch

dev.to
How to Choose 3 Healthtech Email API Paths: Retention Rules That Survive Audits

How to Choose 3 Healthtech Email API Paths: Retention Rules That Survive Audits

dev.to