Memory in an XRPL Node: Explicit Heap Trimming after Cache Sweeps

For operators

• If you measure your node's memory, read the xrpld process, its RSS from /proc/<pid>/status or a process exporter, not the systemctl status figure, which includes the kernel's page cache.
• If you run xrpld 3.3.0 or later, your node already trims the heap after every cache sweep. There is nothing to configure.
• If you build or run xrpld with an allocator other than glibc's, the trim does not apply.
• If your memory graph has become a fine sawtooth since upgrading, that is the trim working, not a fault.

Background

The previous analysis identified the cause of memory growth as allocator behaviour rather than a memory leak. The node's caches released memory as expected at regular intervals, but the process's memory footprint continued to increase. This was due to glibc's allocator retaining freed memory instead of returning it to the operating system. The application was aware of the timing of large memory releases, but the allocator was not informed of these events.

glibc provides the malloc_trim function, which instructs the allocator to return unused memory to the kernel. This post describes the implementation of explicit heap trimming, the criteria for invoking malloc_trim, the trigger points considered, why allocator-level alternatives were not pursued, associated performance costs, and the observed impact during controlled testing and production deployment on Mainnet validators.

The path from finding to release took most of a year. Candidate trigger points were tested on a lab validator in November 2025, the change was merged into the development branch in February 2026 and measured in the performance lab, and it then ran on a Mainnet node through the 3.2.0 release-candidate period. It was held back from 3.2.0 as a precaution while a consensus-stability question was investigated, so 3.2.0 as released does not contain it although its release notes initially listed the change. The investigation found no effect from trimming, and the change shipped in 3.3.0 in August 2026.

The change was introduced in xrpld version 3.3.0 and is now active on all nodes running that release or later. On Ripple's Mainnet validators it lowered steady-state resident memory by 13% to 23%. The following sections detail its operational effects.

Allocator retention heuristics and explicit heap trimming

glibc's memory allocator is designed for efficient repeated allocations by placing freed memory onto free lists, which avoids system calls for similar-sized requests. To reduce lock contention, glibc implements multiple independent heaps called arenas. Each active thread is assigned an arena, and on 64-bit systems, the allocator can create up to eight arenas per processor core (see mallopt(3)). In multi-threaded applications, this results in free memory being retained across many arenas simultaneously.

Left to itself, glibc returns freed memory to the operating system in two scenarios:

• Large allocations. Allocations at or above the M_MMAP_THRESHOLD are given their own memory mapping from the kernel with mmap, and that mapping is removed as soon as they are freed.
• Free space at the top of an arena. Smaller allocations are handled within arenas. An arena releases memory only if contiguous free space at the top exceeds M_TRIM_THRESHOLD, which defaults to 128 KiB. Free memory located elsewhere in the arena is retained regardless of its size. Both thresholds are dynamic. By default, glibc increases the mmap threshold when a block larger than the current threshold is freed, raising it to the size of that block, up to a maximum of 32 MiB on 64-bit systems. The trim threshold is set to twice the mmap threshold. As a result, if a process occasionally frees large blocks, the allocator becomes less aggressive in trimming memory. There is no automatic adjustment in the other direction. The kernel cannot signal memory pressure to the allocator, and the allocator does not revisit previously retained memory.

xrpld's allocation pattern is poorly handled by these heuristics. The heap is dominated by small objects, such as tree nodes and ledger entries, allocated in large numbers into caches with varying lifetimes. When caches evict, freed chunks are interleaved with live allocations, so the top of each arena is rarely free and automatic trimming is infrequent. Freed memory therefore remains resident across many arenas, and the process grows until the retained memory matches the application's allocation churn, the rate at which it allocates and frees short-lived objects. This retained volume was the growth measured in the previous post.

malloc_trim is the allocator's explicit release path. Since glibc 2.8 it iterates over every arena and returns every whole page of free memory to the kernel, wherever that page sits within the heap, using the madvise system call to release pages, or sbrk to shrink the top of the main heap (malloc_trim(3)). It is not limited to the top of the heap, which is what makes it effective where the automatic path is not. Its single argument, pad, reserves an amount of free space at the top of the main heap. Only the main heap honours it. The per-thread arenas ignore it.

There are two main limitations to malloc_trim:

• It does not move live data. Any page containing at least one live allocation remains resident, so the memory cost of fragmentation is reduced but not eliminated.
• It is specific to glibc. Other allocators, such as jemalloc or tcmalloc, do not implement this function, and instead manage memory release according to their own policies. Two alternatives were not pursued in this work. Limiting the number of arenas with MALLOC_ARENA_MAX reduces how many places free memory is retained, at the cost of more lock contention, but it does not return free pages inside an arena, which is where the growth was. Replacing the allocator with jemalloc or tcmalloc changes memory behaviour for every build and deployment at once. An explicit trim at a single call site was the least intrusive change, could be evaluated in isolation, and left the allocator untouched for operators who build differently. An evaluation of jemalloc is under way separately.

malloc_trim is available and can release memory that the automatic trimming path cannot reach. The primary cost is a scan of the heap and the page faults that occur when released pages are subsequently reused. The remaining question is when to call it.

Trigger selection

malloc_trim should be invoked only after the application has released a significant amount of memory. Invoking it at other times pays for a heap scan with little to show for it, and frequent calls compound this cost. The selection criterion was defined as points in xrpld where the application knows that a large release has just completed. Five candidate locations were instrumented in PR #6022, with each call tagged by its trigger and the process Resident Set Size (RSS) logged before and after it. The sweep trigger and the two online-delete hooks ran together for 40 hours on a lab validator connected to Mainnet. The order-book and sync-completion triggers were judged by how often they fire, and the per-cache variant was superseded by the application-level call before the run.

• Inside each cache sweep. The first implementation called malloc_trim from TaggedCache::sweep, once per cache. Several caches sweep in close succession, so this version needed a rate limiter with a five-second minimum interval and a re-entrancy guard to prevent overlapping calls.
• After the application sweep. Application::doSweep executes all cache sweeps sequentially as a background job, with the interval determined by node_size. Invoking malloc_trim once after the final sweep captures all memory releases in the cycle (commit). This trigger accounted for the majority of memory reclaimed, executed at each sweep interval, and reclaimed memory on nearly every call.
• After online-delete rotation. Two hooks in SHAMapStoreImp were evaluated: one following ledger cache clearance and another after old ledger deletion (same commit). Both triggers were infrequent due to the rarity of rotation events, and most memory released during rotation was already reclaimed by the subsequent sweep.
• After the full order-book rebuild. OrderBookDB::update walks the ledger state every 25,600 ledgers, a constant in the code that works out to roughly once a day, and only on nodes with pathfinding enabled (commit). It cannot provide regular reclamation.
• On sync completion. The transition into the full operating mode in NetworkOPs (commit), which happens once after start and then only when the node loses and regains synchronisation. Too rare for regular reclamation, and with a drawback raised in review: a node that has just regained sync is about to begin its heaviest catch-up work, so trimming at that moment means paying to release memory that will be reallocated immediately.

RSS before and after every malloc_trim call, by trigger. Lab validator connected to Mainnet, 11–13 November 2025, 40 hours. Green candles are calls after which RSS was lower, red candles calls after which it was higher. Top: the sweep trigger, firing at every sweep interval. Middle and bottom: the two online-delete triggers, firing a few dozen times over the run.

The decision documented in the PR was based on two factors: the sweep-based trigger achieved the highest observed memory reclamation, and it operates within a background job context, keeping heap scan overhead off the consensus and RPC paths. Only the sweep-based trigger was retained.

The best-performing trigger also determined where the call should be made from. Invoking malloc_trim within each cache's sweep required throttling for safety. A single invocation after Application::doSweep completes all cache sweeps does not require throttling, as the call frequency is limited by the sweep interval and each invocation addresses the full release cycle. The final implementation includes a single call in Application::doSweep, with the rate limiter and re-entrancy guard removed.

Trim padding

The pad argument of malloc_trim reserves free space at the top of the heap so that the next allocations do not have to grow the heap again. A non-zero reserve could in principle reduce the cost of refilling the caches after each sweep. In practice, only the main heap honours pad, and the per-thread arenas that serve xrpld's worker threads ignore it, so a reserve could protect at most the main heap's share of the memory.

Four 12-hour test runs were executed on the lab validator connected to Mainnet. Trimming was invoked from Application::doSweep with pad set to 0, 256 KiB, 1 MiB, and 16 MiB. RSS was sampled at 10-second intervals, and the duration of each trim operation was logged. To eliminate the impact of the initial ramp-up, RSS statistics are reported from the second hour of each run onward.

The differences are small. Mean RSS varies by 0.4 GiB across the four settings, under 3%, and mean trim duration by 16 ms, about 10%. Each setting was run once, in sequence, against whatever Mainnet traffic arrived during its twelve hours, and variation between runs from load alone is of the same order. The comparison can therefore rule out a large effect but cannot rank the settings. A small effect is also what the semantics predict, since only the main heap honours the reserve.

The default was kept. pad is zero, and it is a compile-time constant rather than a configuration setting, so that the change adds no tuning surface to the node.

Memory results

The change was measured in three tests: a lab machine connected to Mainnet, a closed performance network under a fixed transaction load, and two of Ripple's Mainnet validators over the summer.

Lab machine

The lab validator connected to Mainnet ran two builds on the same hardware and settings: the unmodified binary, and a binary with the trim triggers as they stood in November 2025, which called malloc_trim from the application sweep and from the two online-delete points, of which only the sweep call shipped. RSS and Referenced memory were sampled every 10 seconds, and each trim call recorded the process RSS before and after it. The baseline ran for about 24 hours, the trimmed build for about 40, and one run was made of each. The trimmed run is the one shown in the candle charts in the Trigger selection section.

• Baseline. RSS grew from 14.0 GiB to 32.9 GiB over 24 hours and was still rising when the run ended.
• Trimmed. RSS climbed to 16 GiB within the first hour, to 17 GiB by the sixth, and then stayed within an 18–20 GiB band for the rest of the 40 hours, drifting upward at about 30 MB per hour.
• At 24 hours of process life, RSS was 18.3 GiB in the trimmed build against 32.9 GiB in the baseline, 44% lower. The trimmed value is read from the trim log's per-call RSS record, so the two processes are compared at the same age.
• Over the 40 hours, the trimmed build made 1,389 calls, 1,173 of them from the sweep. 91% of them reduced RSS, by a median of 75 MB and a maximum of 1.38 GB, and the observed RSS decrease across all calls summed to about 147 GB, against a net RSS growth of about 14 GB.
• RSS minus Referenced in the trimmed steady state was about 0.2 GiB on a 19 GiB process, about 1%, so almost all resident memory had been accessed since the window opened. ### Performance network

The performance lab runs a closed five-validator network under a fixed transaction load. Comparison tests with and without malloc_trim each ran for five hours. Validator nodes used 2.58 GB less RSS with trimming enabled, a reduction of 14.8%. Client-handler nodes used 3.33 GB less, a reduction of 18.2%. The CPU and consensus-timing measurements from the same runs are in the Overheads section.

Mainnet validators

Two of Ripple's validators connected to Mainnet, one on the default UNL (dUNL) and one not, ran 3.2.0 without heap trimming from 16 June to 31 July 2026 and 3.3.0 with it from 7 August. The trim call was removed from 3.2.0 the day before its release, so the 3.2.0 builds in this comparison do not contain it, although the release notes initially listed the change. The week between the windows saw the 3.2.1 hotfix release and an unrelated network incident, and was excluded. The 3.2.0 windows close before the incident, and leaving out their final three days changes the plateau means by less than 0.05 GiB. Process RSS is recorded by Prometheus from the process exporter, and both windows on both nodes open on a fresh start of the process, so the two versions are compared at the same age. The figures below are the plateau each node settled at after its first five days.

Resident memory of two of Ripple's Mainnet validators, 3.2.0 without heap trimming and 3.3.0 with a trim after every sweep, each window opening on a fresh start and plotted by days since start. Dotted lines are the plateau means after day five. The dUNL node's two-day rhythm is online delete: it rotates its history every 50,000 ledgers, about every two days, and under 3.3.0 each rotation's release leaves RSS instead of staying with the allocator. The non-dUNL node rotates every 4,096 ledgers, every few hours, so its releases are small and its line is smooth.

This is a comparison between versions, and 3.3.0 differs from 3.2.0 in more than the trim call. The reduction is nonetheless attributable to trimming: nothing else in the release targets memory retention, the production graphs show the sawtooth of release and regrowth that trimming produces, and the reduction is in line with the lab and performance-network tests, where malloc_trim was the only difference between the test runs compared.

The Mainnet validators test has the most weight of the three, since these are production nodes. The windows are 44 and 38 days long, the savings appear on both nodes, and they are well above the ordinary drift either node shows over a window of that length.

Overheads

Trim latency

The MallocTrim helper was instrumented to measure the cost of each call. With debug logging enabled, the helper records, for every invocation, the process RSS before and after execution and the wall-clock duration.

On the lab validator connected to Mainnet, a sweep trim took a median of 141 ms and a 95th percentile of 340 ms, at roughly one call per minute, which is a fraction of a per cent of the sweep interval, spent on a background thread. The four 12-hour padding runs, with 620 to 706 sweep trims each, gave the same picture, with mean durations of 163 to 179 ms. The duration of a call grows with the size of the heap and with the amount of memory freed since the previous sweep, so nodes with a different node_size or online_delete configuration may see longer calls.

The trim operation executes as part of the background sweep job, once all cache sweeps have completed and outside any cache lock, running on the sweep thread. In glibc, malloc_trim acquires each arena's lock sequentially during trimming. Threads attempting allocation or free operations on a locked arena are blocked until the lock is released. The following subsection presents consensus-timing measurements to assess the practical impact of these waits.

CPU and consensus timing

In the performance lab, a five-validator network was used to execute comparison tests with and without malloc_trim, each for five hours under a fixed transaction load. CPU usage on validator nodes increased by 0.7% with trimming enabled, while client-handler nodes showed a 3.4% decrease. Both changes are small. Consensus timing was evaluated using Cliff's delta with Benjamini–Hochberg correction across all metrics. No changes were detected in ledger close interval, its 80th percentile, voting duration, or transaction-set acquisition time. Per-transaction apply, flush, and build times showed statistically significant differences, as expected given the number of ledgers processed, but all effect sizes were below the 0.147 threshold considered negligible by this method. The test was repeated several times with consistent results, and the figures above are from one round.

Stability

Two checks were made on whether trimming affects a node's participation in the network.

The first comparison was made during the 3.2.0 release-candidate period, when a Mainnet node ran candidate builds with the trim call for 22 effective days and a candidate build without it for 4.65 days. Departures from the full state occurred at rates of 2.41 and 2.15 times per day, with median durations of 3.16 and 3.13 seconds, respectively. Consensus step-outs were observed at 2.23 and 2.15 times per day. None of these differences, nor the difference in daily counts of SHAMapMissingNode events, was statistically significant. The comparison is limited by the short duration of the window without the call, so only large effects can be excluded.

The second check was conducted as part of an investigation into slow initial synchronisation on Devnet during the 3.3.0 release candidates, where new nodes required 36 to 59 minutes to reach full synchronisation. Trimming was evaluated as a potential cause. In the slowest of three instrumented runs, trimming accounted for 305 calls totalling 72 seconds during a 59-minute sync, representing about 2% of wall time, with approximately 19 seconds occurring within the synchronisation window. Sync time did not vary with sweep interval settings of 60, 600, or default values, and trim pauses did not align with the acquisition timeouts that defined the slow runs. The investigation found no evidence that trimming contributed to slow sync times. The causes of the slowdown lay elsewhere and are outside the scope of this post.

Outstanding questions and next steps

Introducing explicit heap trimming after the cache sweep consistently reduced resident memory usage on xrpld nodes across all measured environments: by 13% to 23% on Ripple's Mainnet validators, by 15% to 18% in the performance network, and by 44% on the lab machine under the heavier load of late 2025. The call cost a fraction of a per cent of the sweep interval on a background thread, and had no measurable effect on consensus timing or node stability. This change was released in version 3.3.0, implemented at a single call site without additional configuration parameters.

The observed reduction in resident memory varied widely: 23% and 13% on the two Mainnet validators, 15% to 18% on the performance network, and 44% on the lab machine. This variation exceeds what can be explained by measurement differences alone and requires further investigation. One hypothesis fits the evidence so far: the reclaimed memory is memory the allocator retained after short-lived allocations were freed, so its volume follows the rate of allocation churn, which in turn follows transaction load. The largest reduction was recorded in November 2025, during a period of network-wide memory growth, while the Mainnet measurements were made in the summer of 2026. If transaction volume or composition during that period drove the growth of retained memory, the saving would shrink as load normalised. The stabilisation of untrimmed validators at a plateau is consistent with this hypothesis. Verification will require historical correlation of network load with memory metrics and controlled lab experiments with varied transaction mixes.

Several areas for further optimisation have been identified:

• Trim frequency. Performing trimming on every sweep is straightforward but may not be the best schedule. Adjusting the schedule to trim every few sweeps, or only when a sweep releases memory above a defined threshold, could reduce the number of trimming operations while retaining most of the memory savings.
• Padding. The initial four single runs excluded a large effect from pad and did not allow for ranking of different settings. Conducting repeated, interleaved runs is necessary to determine whether maintaining a small memory reserve provides measurable benefit.
• Residual growth. The gradual increase in memory usage observed with trimming enabled reflects the live working set, which cannot be reduced through allocator configuration. Addressing this requires revisiting the cache experiments from part one and re-running them in the updated lab environment with trimming enabled.
• Allocation churn at the source. Trimming returns memory after the application has freed it. It does nothing to reduce how much the application allocates and frees in the first place, and that churn is what causes the fragmentation the trim works around. The complementary work is to find which code paths allocate the most, using the same profiling as in part one, and to reduce it where possible: allocating fewer short-lived objects, or reusing them instead of recreating them.