One Abuse Complaint Later: Hardening a Home Proxy Without Blocking All UDP

This piece continues a previous write-up on diagnosing injected TCP resets [1]. That one was about proving where a connection failed. This one is about keeping a shared exit within its provider's rules and recoverable when its own tooling fails.

Part 1. The incident: forty minutes to first complaint

The setup is boring on purpose. One cheap virtual server, a gigabit port, a proxy core (Xray) with a REALITY inbound, and a house full of devices that route through it by domain rules on the router. Nothing exotic; the build takes an evening and about twenty dollars a year. It served video, cloud sync, and late night music discovery before this incident.

Then a family member installed a torrent client.

The client did what torrent clients do. It announced itself to public trackers, joined the public DHT network through its bootstrap nodes, and started moving gigabytes at line rate. Nobody in the house noticed anything; the video streams were fine, the ping was fine, everything was fine.

Forty minutes later the hoster forwarded the first abuse complaint. The message was professionally assembled: a timestamp with a timezone, the exact protocol their sensor had matched, both endpoints of the observed flow, and a link to the acceptable use policy. The complaint documented a flow associated with the exit address. I could not reconstruct the monitoring system behind it from that notice alone.

The policy page left no room for creativity. Repeat complaints end with the address null-routed, and null-routed budget servers do not get appeal processes. The exit had to become hostile to BitTorrent, and it had to happen before the next evening, because torrent clients tend to run on schedules. Whoever starts the client, the operator of the exit answers for what leaves it.

The instinctive fix, and I almost shipped it, reads one line long: block UDP. The reasoning is sound as far as it goes. The DHT and uTP use UDP, so blocking UDP removes important paths. It does not stop TCP peer connections or every tracker.

In my incident, observed torrent traffic stopped. Several unrelated applications broke too.

Part 2. What blanket UDP blocking actually breaks

The first attempt dropped every UDP flow on the exit except DNS on port 53. Torrent traffic stopped within minutes, which looked like victory for about an hour.

Messenger calls were disrupted while text messages continued to work. WhatsApp survived through TCP fallback. Blocking UDP did not make every call impossible; it made the household depend on whichever fallback each application could manage.

The split between messages and calls was the useful clue. A service could remain reachable while one of its communication paths was blocked. I removed the blanket rule and tested calls again.

QUIC was affected too. I did not measure a browser-wide latency penalty. Games and browser videoconferencing belong in the regression checklist; I did not keep controlled results for them.

That is the trap with a transport-level block: ordinary browsing and messaging can make the connection look healthy while other functions fail. A dashboard of established TCP sessions would not establish that UDP-dependent traffic still worked.

The lesson cost one evening. Blocking an entire transport was too broad for accidental BitTorrent use on this shared exit. The replacement needed narrower rules and tests of the household applications that mattered.

Fig. 1 - selective filters cover some discovery paths; cached nodes and direct peers can bypass them

Part 3. The insight: block peer discovery, not the protocol

A client needs a way to discover peers before it can transfer data. Common starting points include trackers, DHT bootstrap nodes, and information already stored locally.

These sources overlap, but they are not a closed list. Trackers may use hostnames or IP addresses. Clients can retain peers and DHT routing tables across restarts. Torrent files may contain node addresses, and users can supply peers directly.

BEP 5 explicitly describes persistent routing tables and embedded nodes. A hostname deny list cannot remove that knowledge from a client.

The objective here is narrower: reduce the discovery paths available to ordinary household clients, while avoiding a blanket UDP block. This is a mitigation for accidental use, not a guarantee against another complaint. Where the hoster's policy requires reliable prevention, endpoint controls or keeping untrusted devices off the shared exit remain necessary.

Translated into ordered routing rules on the exit node:

[ { "type": "field", "protocol": ["bittorrent"], "outboundTag": "block" }, { "type": "field", "domain": [ "geosite:category-public-tracker", "router.bittorrent.com", "dht.transmissionbt.com", "dht.libtorrent.org", "router.silotis.us" ], "outboundTag": "block" }, { "type": "field", "port": "6881-6889", "outboundTag": "block" } ]

Four checks are expressed as three routing rules: the protocol sniffer, tracker list, bootstrap names, and port range. The tracker list and bootstrap names share one domain rule.

These rules have two prerequisites. Enable traffic sniffing on every inbound that carries the traffic being filtered, and provide the block outbound. Merge this field into each existing inbound, keeping its listener, credentials, and transport settings:

{ "sniffing": { "enabled": true, "destOverride": ["http", "tls", "quic"], "metadataOnly": false, "routeOnly": true } }

Add this object to the existing outbounds array if the tag is not already defined:

{ "tag": "block", "protocol": "blackhole", "settings": {} }

The snippets are configuration fragments, not a replacement configuration. Keep the ordinary outbound first: it remains the default for unmatched traffic. Without enabled payload sniffing, the BitTorrent protocol rule cannot identify its handshake. bittorrent is not a destOverride value; that list controls supported hostname overrides. Here, routeOnly: true uses the sniffed hostname for routing without replacing the connection's destination address. If the inbound already has a sniffing policy, reconcile it rather than replacing it blindly.

The protocol sniffer catches plain BitTorrent handshakes on any port. Honesty matters here, and the documentation is refreshingly blunt about it: the built-in sniffer is basic and may miss encrypted or obfuscated handshakes. Treat it as a tripwire, not a wall. Its usefulness depends on the handshake that actually reaches the sniffer.

The tracker list is a community-maintained geosite category. Its coverage changes, so verify the domains relevant to your clients instead of assuming it includes every default tracker.

The four DHT names in the rule above are useful bootstrap targets to cover, not a universal or mandatory bootstrap set. Other names, direct addresses, saved routing tables, and embedded nodes can bypass them.

The missing prerequisite: a visible hostname

A domain rule needs a domain to match. DNS resolution on a laptop does not automatically reveal that name to the exit server. What matters is what the proxy receives or can recover.

What reaches Xray	Can a domain rule help?	Important limit
A proxy destination containing a hostname	Yes, if that name matches the rule	Confirm the client actually sends the hostname
An IP destination with a supported name-bearing handshake	Sometimes, with suitable sniffing	Inspect the configured sniffer and routing behavior
Raw DHT UDP sent to an IP address	Not from a hostname list alone	The packet need not contain the bootstrap name
A cached peer or manually entered address	Not from its earlier DNS history	Other rules may still match the flow

That distinction explains a frustrating test result: the list can contain the right hostname while the connection still passes. Adding more names will not fix missing name visibility.

The installer below is idempotent. It cleans its tagged rules before writing replacements, including the blanket-UDP attempt from Part 2. Re-running it does not duplicate its tagged rules. The marker uses Xray's documented ruleTag field. Review the two legacy shapes before using it: exact equality cannot prove who originally created a rule. The installer assumes the inbound sniffing settings and block outbound are already configured.

https://gist.github.com/Vitaliy69/26606b08a805eb0144a82d0392249f3a?embedable=true

Run the installer with permission to preserve the original file's owner and group. My service reads a root:xray configuration with mode 0640. Copying only that mode to a root-created temporary file can leave it owned by root:root, making the replacement unreadable to the service. The snippet preserves ownership before replacement and creates the temporary file privately. A successful validation as root does not prove that the service account can read it; check the configuration under the actual service user before restarting.

Testing covered three properties, because duplicates are not the only way to fail. A double run produced no duplicated rules, because cleanup now keys on the ruleTag and cannot miss its own output. A scratch config carrying a foreign UDP rule and a foreign bittorrent-shaped rule kept both untouched: only tagged rules and the two known pre-tag shapes are removed. The write path goes through a backup, a temporary file, an Xray configuration test, and an atomic replace. The code writes the file; applying it to the running service is a separate step. An earlier version matched every UDP rule as removable, and the one after it read one field while writing another, creating a duplicate per run. Both mistakes stay in this paragraph as a warning: installers are code, and code needs tests.

One more bug survived the test doubles. Xray infers the configuration format from the filename unless told otherwise; config.json.tmp has an unsupported extension. My stub returned success without checking that. On Xray 26.3.27 for macOS arm64, the old installer failed with "Failed to get format" and left the original file untouched. The correction explicitly selects JSON. The final snippet, including mkstemp and ownership preservation, then passed two runs as root with real Xray 26.3.27 for Linux arm64 in a disposable container. Both runs preserved foreign rules and root:xray ownership with mode 0640; validation also passed as the xray user. An invalid configuration was rejected without replacing the original or leaving a temporary file. These checks exercise configuration loading and file access, not live traffic filtering.

Test the policy, not just a timeout

Choose a tracker URL that answers through the unfiltered route. Use its real protocol; an HTTPS request to a UDP-only tracker is not a useful control.

# Set TRACKER_URL to a known-working HTTP(S) tracker endpoint. # Keep this shell variable quoted. A URL may contain an ampersand. curl --max-time 8 --silent --show-error --output /dev/null "$TRACKER_URL" rc=$? printf 'curl exit=%s\n' "$rc"

A timeout or TLS failure alone does not identify the blocking component. Compare the same endpoint with the relevant rule disabled, then enabled, and correlate the result with routing logs. Run a reachable control request too. Do this in a controlled test window, using authorized content and a test device.

The following matrix makes the boundary visible before trusting the policy:

Test	What to observe	What it does not prove
Fresh client, selected tracker and bootstrap paths	A rule match and failed discovery on those paths	Every possible tracker is blocked
Same client with retained peers or DHT state	Whether transfers resume despite hostname filters	A previous cold-start test was invalid
Call or QUIC session through the same exit	Whether unrelated traffic still works	Every application or UDP port is unaffected
Known direct peer on a custom port	Whether another rule catches it	Hostname filtering can see raw IP traffic

Record the Xray version, client version, rule order, and whether the client state was fresh. A screenshot of an idle torrent client omits all four.

Do not pipe curl into head and treat the pipeline status as curl's result. Without pipefail, a successful final command can hide an earlier failure. Capture the status of the command being tested directly.

Part 4. Rejected alternatives, for the record

Four other designs got considered and rejected before the layered approach, and the reasons are instructive.

Blocking torrents on the router by DNS failed for two reasons. First, devices with hardcoded DNS or DNS-over-HTTPS bypass the router resolver entirely, and the exit cannot assume every device uses its resolver. Second, DNS filtering says nothing about direct IP connections, and DHT speaks raw UDP to addresses.

Per-device policing, the management answer, does not survive contact with guests and phones. It also pushes the problem onto people who cannot debug it, which converts an engineering task into a series of arguments.

Port-based blocking alone, without the domain layers, fails because modern clients randomize their listening ports on first run. The 6881-6889 range stays in the rule set anyway, as a cheap belt under the suspenders, but nothing depends on it.

Rate limiting torrent-shaped traffic was the most tempting sophisticated option and the easiest to reject. Shaping requires classifying the flow, which returns to the encrypted-handshake problem, and slower traffic can still participate in a monitored swarm. Lower transfer speed does not eliminate the risk of an abuse complaint.

Part 5. A note on transports

Some client networks, mostly mobile, drop long-lived connections. On those paths the TCP profile sometimes failed in a recognizable pattern: a TLS handshake error, then timeouts, while ICMP still answered. A second inbound using the XHTTP transport in stream-one mode, on the same routing table, worked where the TCP profile failed in my tests. I did not determine why the paths behaved differently; the second inbound is a comparison path, not a diagnosis.

The full transport story, including the sniffing trap that silently disables domain rules when clients resolve locally, belongs in its own write-up. Two details travel with this article because they cost an hour each: in this tested configuration, Vision belongs on the TCP profile; the XHTTP profile omits that flow flag. Sniffing can recover supported names when a client sends an IP destination. Connections already addressed by hostname do not need sniffing for that name to be available.

A later experiment with an additional TLS endpoint exposed another test boundary. Server-side checks passed: the publicly trusted certificate validated, and the listener accepted requests. From the household client, traffic still stalled; an independent client reproduced the failure. The same request sent through an SSH forward to the same listener completed. That comparison placed the failure on the direct network path rather than in the server configuration, without identifying its cause. I removed the experimental endpoint. A server-side smoke test had never proved that the household could use it.

Part 6. The case of the corrupted updater

A defense built on domain lists is only as fresh as the lists, so the exit needs a geo database update job. This section exists because mine failed in the most educational way possible.

An installer download returned HTML instead of shell source. My preserved setup guide used the form bash -c "$(curl ...)": command substitution puts the response into a single command-line argument. A large response can exceed the operating system's argument limit before Bash parses any of it. That is consistent with the recorded "Argument list too long" failure. Piping a response into Bash's standard input is a different mechanism; it does not hit that argument-size limit.

The recovery lesson was to stop treating a downloaded response as executable source before checking it. The smaller updater below fetches only data files, validates the configuration that uses them, and attempts to restore the old files if replacement fails.

The following Linux example updates databases only. It does not update the Xray binary or systemd unit. It assumes both old database files exist and runs with permission to replace them and restart Xray. It requires flock, supplied by util-linux on Ubuntu.

set -euo pipefail exec 9>/run/lock/xray-update.lock flock -n 9 || { echo "Another Xray update is running" >&2; exit 1; } DAT=/usr/local/share/xray tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT curl -sfLo "$tmp/geosite.dat" \ https://github.com/v2fly/domain-list-community/releases/latest/download/dlc.dat curl -sfLo "$tmp/geoip.dat" \ https://github.com/v2fly/geoip/releases/latest/download/geoip.dat [ "$(stat -c%s "$tmp/geosite.dat")" -gt 1000000 ] || { echo "geosite too small"; exit 1; } [ "$(stat -c%s "$tmp/geoip.dat")" -gt 500000 ] || { echo "geoip too small"; exit 1; } for f in geosite geoip; do head -c1 "$tmp/$f.dat" | grep -q '<' && { echo "$f looks like HTML"; exit 1; } done cp "$DAT/geosite.dat" "$DAT/geosite.dat.bak" # rollback point cp "$DAT/geoip.dat" "$DAT/geoip.dat.bak" rollback() { # defined BEFORE first risk local failed=0 for f in geosite geoip; do if ! cp -p "$DAT/$f.dat.bak" "$DAT/$f.dat"; then echo "could not restore $f; backup retained" >&2 failed=1 fi done if [ "$failed" -ne 0 ]; then echo "incomplete database restore; investigate manually" >&2 exit 1 fi if ! xray run -test -c /usr/local/etc/xray/config.json >/dev/null; then echo "files restored, validation failed; investigate manually" >&2 exit 1 fi if systemctl restart xray && systemctl is-active --quiet xray; then echo "files restored; service active; verify listeners and a client request" >&2 else echo "files restored, service recovery failed; investigate manually" >&2 fi exit 1 } install -m 644 "$tmp/geosite.dat" "$DAT/geosite.dat" || rollback install -m 644 "$tmp/geoip.dat" "$DAT/geoip.dat" || rollback xray run -test -c /usr/local/etc/xray/config.json >/dev/null || rollback systemctl restart xray || rollback # restart itself refused sleep 1 systemctl is-active xray >/dev/null || rollback

The lock is acquired before downloads, backups, or replacements. A competing invocation exits without touching the shared backup files. Every updater that writes these databases must use the same lock path. The shell keeps file descriptor 9 open through recovery and releases the lock on exit; keep the lock file in place rather than deleting it. A different updater that ignores this lock can still race with the example.

In the Linux container test, a held lock rejected a competing invocation before any download, backup, or service command. With the lock free, the updater completed and released it on exit. The test used real flock and Xray validation; downloads and service-control commands were test doubles.

An active process is the first health check, not the last. Confirm every expected listener with ss -lntp, using the ports in your configuration. Then send a control request through the actual client profile. A local request on the VPS bypasses the inbound and cannot substitute for that test. These checks remain separate from the small database-update script.

The -f flag makes curl fail on HTTP errors, explicit failure branches cover the mutation steps; set -e provides additional exit behavior with shell-specific exceptions, the size floors catch truncation, and the first-byte check rejects a leading <. It misses HTML with leading whitespace and many other invalid payloads. None of that proves the database is valid, so the pipeline ends by asking Xray itself: config validation, restart, health check, and a rollback to the saved copies if any step objects. The updater attempts to restore the previous databases and restart Xray. Both restore copies are attempted, and backups are retained. An incomplete restore stops before restart and reports that manual intervention is required. Claiming more than that would be selling the guards past what they do.

The meta-lesson generalizes past geo databases: an update pipeline is a threat surface against your own infrastructure, and it deserves the same paranoia as a public endpoint.

A rollback message is not a rollback test

The original failure taught me to test the updater's recovery path separately from a normal update. Replacing curl, systemctl, and the installer with test doubles lets a local harness force failures without changing a server.

There are three separate questions. Did the script exit with failure? Did it restore the previous files? Can the service start with those files? A passing answer to the first says nothing about the other two.

For the database-only example, inject a failure into each install, configuration validation, restart, and the active-state check. Compare restored file contents with the saved originals. Also fail each restore copy independently. Verify that both copies are attempted and the backups remain. Fail recovery validation and restart separately; each must report failure without claiming recovery.

A full binary updater needs more tests. Preserve executable permissions, unit files, and the previous presence or absence of installer-created paths. A backup command returning success is not enough; verify that the expected backup exists and contains the original data.

These are suggested release checks, not claims that a stubbed service models a live kernel. A local harness can prove script branches and file restoration. It cannot prove that an upstream archive, service manager, or network behaves exactly like the stub.

Test the snippets extracted from the article itself. A passing test of a different working copy does not verify what the reader will run.

Part 7. A closed TCP state did not mean released memory

Correct routing rules were not the end of hardening this exit. A later slowdown exposed a different limit: Linux was under TCP memory pressure while hundreds of megabytes of system RAM remained available.

Selected destinations then used a separate process: Xray forwarded them to warp-svc through a loopback SOCKS listener. An October 27 audit of WARP 2026.8.2115.0 found 33 socket descriptors in TCP_CLOSE holding 115.26 MiB, plus six sockets in FIN_WAIT2 holding another 28.59 MiB. Global TCP memory was 144.72 MiB. Two snapshots thirty seconds apart showed the same allocations; some half-closed connections had retained data for over forty minutes.

TCP_CLOSE described the protocol state. The process still held descriptors referring to those sockets, and their buffers remained charged. The initial ss output did not account for all of it. Following the process's descriptors, duplicating them with pidfd_getfd, and querying SO_MEMINFO and TCP_INFO found the missing allocations. The audit did not read payloads or close the original descriptors.

Restarting only WARP, without changing Xray or its configuration, reduced global TCP memory from 144.72 to 6.11 MiB. A new half-closed WARP socket soon retained about 5 MiB again. The restart identified the process holding the buffers; it did not establish the internal defect or eliminate its cause. Nor was tcp_fin_timeout a general cleanup switch: the Linux documentation defines it for orphaned FIN_WAIT_2 connections, whereas this socket still had an application reference.

A bounded reproduction bypassed Xray and used WARP's SOCKS listener directly. It mixed completed uploads, cancelled downloads, and uploads whose destination replied before consuming the body. Early-response cases retained buffers thirty seconds after completion in two of three runs. The third retained none at that checkpoint. All three actually used H3, even when warp-cli displayed an H2 setting. Requesting a transport had not proved it was in use.

An isolated replacement test used Xray 26.10.21's native MASQUE outbound with verified H2. After six rounds totaling thirty requests, the test instance's cgroup memory fell to 9.93 MiB two minutes after the workload ended. Implementation and device registration changed too, so this was not a controlled H2-versus-H3 comparison.

After migration, the old loopback hop was gone. Absence of its sockets could not prove that the replacement released its own buffers: native MASQUE uses a userspace network stack. Roughly forty-six hours later, the production Xray instance's cgroup held 25.08 MiB, including 23.45 MiB of anonymous memory, with a peak of 187.01 MiB of unknown timing and cause. That cgroup peak and the old kernel socket allocation measure different things. Four additional TCP-pressure episodes totaled about 6.6 seconds; their cause was also unknown. The snapshot showed memory had fallen, not permanent leak elimination.

A successful response establishes delivery. Hardening also means checking what remains allocated after completion, cancellation, and an early response. Otherwise, a correct traffic policy can still leave the next evening's allowed traffic competing for resources held by finished requests.

Scope: this article covers running a personal proxy within a hosting provider's acceptable use policy. Check your provider's terms and the laws that apply where you and your users are before operating one.

Takeaways

Block the common paths to peers, not the transport. Trackers plus the four bootstrap names cut the initial discovery paths verified here; warm DHT caches and embedded node lists remain, and UDP is not denied wholesale. The port-range rule still blocks matching UDP and TCP destinations.
A protocol sniffer is a tripwire with documented limits. Say so out loud, in the article and in the design review, because overclaiming filters is how incidents repeat.
Blanket UDP denial can disrupt applications even while messages and web pages work. Test calls explicitly; WhatsApp TCP fallback survived in this incident.
Transport choice is a property of the network, not of the server. A second inbound is a comparison path; test it on the networks that matter.
Domain rules need a visible name, supplied by the client or recovered by supported sniffing. Sniffing cannot reconstruct a hostname from arbitrary raw-IP traffic.
Update pipelines are attack surface against yourself. Failure flags, size floors, a first-byte check, and validate-before-restart, all four, every time.
An exit code is a hint, not proof. Pair the failing request with a control host and a routing-rule hit in the logs before declaring victory. Check which command produced the exit status.
Check resource release after requests end. A closed TCP state does not establish that the process released its descriptor or buffers; compare kernel socket allocations and userspace memory separately.