For operators

  • If you measure your node's memory, read the xrpld process, its RSS from /proc/<pid>/status or a process exporter, not the systemctl status figure, which includes the kernel's page cache.
  • If you run xrpld 3.3.0 or later, your node already trims the heap after every cache sweep. There is nothing to configure.
  • If you build or run xrpld with an allocator other than glibc's, the trim does not apply.
  • If your memory graph has become a fine sawtooth since upgrading, that is the trim working, not a fault.

Memory ballooning

Memory growth in XRPL nodes was observed in late summer 2025. Operators reported increasing memory usage on their validators that did not decrease, including on nodes configured identically to others that were unaffected, and across different hardware platforms. The cause turned out not to be a memory leak but the memory allocator retaining memory the application had already freed, and a fix shipped in xrpld 3.3.0. This post describes how that was established.

A memory regression identified in one release during that period was addressed within weeks, but this had no observable impact for operators. The observed memory behaviour existed before the regression and persisted after the fix. The underlying issue remained unresolved at the time. Operators monitoring node caches asked why memory did not come back after a cache sweep, the routine pass that clears expired entries.

Early in the community discussion, it became clear that consistent measurement was essential. Some reports referenced memory usage from systemctl status, which reflects the memory of the service's cgroup. A cgroup is the kernel's resource-accounting container that systemd wraps around every service. In contrast, per-process readings from /proc/<pid>/status or smaps report only the process memory. These two metrics often differed by several gigabytes due to the kernel's page cache. When xrpld (formerly rippled), the XRPL node software, reads from or writes to its database files, the kernel caches these file pages in memory. This cached memory is not included in the process's Resident Set Size (RSS), the kernel's count of pages held in RAM by the process, because xrpld's database library, NuDB, uses standard file I/O rather than memory mapping. The kernel charges this cached memory to the service's cgroup because xrpld initiated the I/O. On busy nodes, the page cache can reach several gigabytes, increasing as the node accesses its database and decreasing when the kernel reclaims memory or when online delete removes old database files. These fluctuations can occur within minutes and are not controlled by the application. As a result, the same machine can report very different memory usage figures depending on the measurement method, leading to differing interpretations by operators.

On a dedicated xrpld machine, it may appear that the distinction between process memory and total memory usage is irrelevant, since it is all the node's memory in the end. However, the two types of memory are managed differently. The kernel controls the page cache, which grows to use available RAM and causes the system-wide memory figure to reflect total machine capacity rather than the process state. In contrast, the process's own memory cannot be dropped by the kernel, only swapped, and it is the only part that code changes can affect. Per-process memory is therefore the right measure of the application's own behaviour, and all subsequent analysis uses it. RSS is the coarsest of these, since it counts every resident page whether or not the process still touches it. The next section adds a finer one, the working set, which counts only the memory the process has actually accessed over a period.

A memory leak was the first suspicion. Agreeing on how to measure memory made it possible to test that suspicion against the alternatives: normal cache operation, a feature retaining more state than expected, or an issue below the application layer. Without the ability to answer this reliably, for any xrpld version and whatever mix and volume of traffic the node handles, using trusted metrics, any conclusions would be speculative.

A toolset was assembled to collect and analyse the relevant measurements. The following analysis describes the investigation, the findings, the attempted fixes, and the eventual realisation that the root cause was located in a different layer of the stack.

The toolset

The investigation focused on three key questions, with the requirement that no conclusion would rest on a single tool.

  • What are the primary sources of the increased memory usage?
  • What mechanisms are responsible for the observed memory growth on some nodes but not others? For instance, which operational flows, features, or traffic patterns trigger it?
  • How reliable are the measurements under real-world conditions? This had to be answered before the other two meant anything. The previous section established that the focus is on process memory, not system-wide memory. The remaining concern is the reliability of each measurement tool. Each instrument has limitations: RSS can be affected by glibc allocator behaviour, flame graphs may attribute bytes to generic frames if stack traces are incomplete, and the kernel's Referenced bit provides only an approximation. Using multiple tools and cross-verifying results is necessary to ensure accuracy.

The approach uses three measurement methods, with visualisation applied to the collected data.

  • A procfs-based working-set tool, specifically Brendan Gregg's wss.pl, uses /proc/<pid>/clear_refs and /proc/<pid>/smaps to estimate Working Set Size (WSS) over a defined window. This tool is integrated with a custom sampling loop that timestamps each reading and records gross totals. It is a low-overhead, always-on, coarse-grained measurement.
  • eBPF/BCC memleak traces allocator calls using uprobes, which are dynamic tracepoints on user-space functions, with aggregation performed in the kernel. This method is lightweight and provides call stacks for outstanding allocations.
  • A custom glibc shim, implemented as an LD_PRELOAD library, is inserted by the dynamic loader ahead of libc to intercept every allocator call. It interposes on malloc, free, calloc, realloc, posix_memalign, and related functions to capture each allocation with a backtrace. Unlike the sampling tools, it records every allocation and every free, so it can answer questions about individual allocations that aggregated data cannot. The cost is a several-fold slowdown and a dedicated run per question, so it serves as the ground truth rather than as a monitoring tool.
  • Flame graphs are generated using Brendan Gregg's FlameGraph toolkit, with input from the eBPF data, to visualise and analyse call-stack data. These tools represent a trade-off between overhead and detail. The investigation begins with coarse, low-overhead measurements, and more detailed analysis is performed only when initial results indicate where to focus.

Layer 1: procfs-based WSS

The primary objective is to determine whether the additional resident memory is memory the process keeps accessing, or memory it holds but has not touched for some time, such as cache entries that were filled once and never read again. RSS alone cannot make this distinction, as it counts all pages in RAM regardless of recent access.

The Linux kernel maintains a per-page Referenced bit, which is set when a page is accessed and can be cleared on demand. wss.pl uses this mechanism by clearing all bits, waiting for a specified interval, and then measuring how much of the process's memory has been accessed during that period, representing the working set for the window. In cumulative mode (-C), the tool clears the bits once and observes Referenced increasing toward RSS as pages are accessed. Only the clearing operation is costly, so sampling frequency can be adjusted as needed.

RSS vs Referenced over 11.5 hours, cumulative working-set view

Eleven and a half hours of the ballooning on our System76 lab machine, September 2025 — wss.pl in cumulative mode, 60 s samples. RSS climbs at ~0.74 GiB/h — spikes partially recover, but the floor only rises. Referenced converges toward RSS — the growth is being accessed, not sitting cold — and the band between them narrows from ~9 GiB to ~3.8 GiB.

The key metric is the difference between RSS and Referenced, representing resident memory that has not been accessed since the start of the measurement. This includes caches, fragmented allocator heaps, and retained but idle structures. The gap narrows over the run as Referenced approaches RSS, indicating that most held memory has been accessed since the start of the measurement. However, the Referenced bit does not distinguish between untouched pages containing live but inactive data and other types of memory. Accurately identifying the nature of this gap is central to the investigation.

Leaks associated with growing live structures, such as caches, queues, or in-flight buffers, result in Referenced increasing in parallel with RSS, since the leaked memory continues to be accessed. Cold leaks, where memory is allocated but the pointer is lost and never accessed again, appear only as an expanding gap and cannot be distinguished from other untouched memory using this method. This limitation is one reason WSS is not used in isolation.

Two caveats:

  1. The clear_refs mechanism provides only an approximate measurement. The kernel traverses page tables to set and read the Referenced bits, introducing a small amount of latency during sampling, around 5% according to Gregg's documentation. Excessive sampling can interfere with the kernel's page-aging information and affect memory reclamation decisions. To mitigate these effects, long sampling intervals are used, and measurements are conducted primarily in a controlled environment.
  2. Tracking memory utilisation using this method is considered experimental and is documented as such in sources including LWN and Brendan Gregg's WSS post. This approach provides a baseline measurement rather than a definitive conclusion.

Layer 2: eBPF + BCC memleak

WSS indicates that memory usage is increasing but does not identify the specific code locations responsible. To obtain this information, BCC's memleak tool is used.

memleak attaches uprobes to the allocator entry points in libc for a specific process. Each allocation is recorded with its size and stack trace, and each free operation removes the corresponding record. After a period of execution, memleak reports the call stacks associated with the largest outstanding allocations. By sampling these reports at regular intervals, it is possible to track how much memory each subsystem has allocated and not yet freed over time.

This approach is feasible for production workloads because BCC aggregates data within the kernel, transmitting only summary information to user space. Streaming every malloc event instead would produce more data than could be collected without slowing a process that allocates as frequently as an XRPL node. Testing showed no measurable impact on node throughput. For accurate stack traces, builds must include debug symbols and frame pointers. Without them, call stacks may be incomplete.

For analysis beyond the largest allocation sites, the byte-attributed data is processed with flamegraph.pl. The width of each box represents the number of bytes attributed to that stack, with the largest boxes indicating the heaviest allocation sites. This visualisation enables rapid identification of the heaviest allocation sites.

Layer 3: the glibc shim

eBPF aggregates allocations by call stack, which identifies the subsystem responsible but not the individual allocation, its size, or the order in which allocations and frees occurred. The glibc shim records exactly that, every allocation and free with a backtrace, at the cost of slowing allocation-intensive code several-fold. It is therefore reserved for targeted runs that answer a specific question about individual allocations. The remainder of this analysis did not need that level of detail, so only the first two layers are used.

Following the bytes: TaggedCache

The SHAMap, which maintains the ledger state, does not manage the lifetimes of its nodes directly. This responsibility is handled by the TaggedCache, a partitioned, expiry-based cache that supports the TreeNodeCache (for tree nodes), the FullBelowCache (tracking locally complete subtrees), and several additional structures. Profiling a live node with memleak identified the TaggedCache subsystem as the primary consumer of outstanding memory. Key components include canonicalize, which consolidates node copies into a single shared instance, the refresh loop (freshenCaches) that copies cached entries to the new database file during online delete, and sweep threads responsible for evicting expired entries. Most of the remaining memory usage was attributed to the creation of SHAMap nodes and items managed by the cache, including fetch-and-deserialise operations and inbound ledger-data handlers. The node was configured with node_size set to medium, which determines cache sizes in xrpld, and online delete was enabled.

Flame graph: outstanding heap bytes at peak, cache-related stacks highlighted

Peak outstanding allocations were measured on a test-lab validator connected to Mainnet in September 2025. Here, 'outstanding' refers to memory allocated after memleak was attached and not yet freed, representing allocations since tracer attachment rather than the process's total memory usage. The graph displays the total outstanding memory as its full width, with each category's share indicated by horizontal width: dark for TaggedCache machinery, light for cache-managed objects, and grey for other allocations. Stack traces were recoloured by category for clarity. The original flame graph is available for reference.

The data was collected with node_size set to medium, indicating that the observed memory usage was not due to unusually large cache configurations. This pattern persisted across varying traffic levels. In all peak snapshots of outstanding memory, 84–99.7% was attributed to cache operations, primarily under SHAMapStoreImp::freshenCaches, which re-populates the new database file from cache keys during online delete. Transient memory spikes were also associated with cache maintenance, proportional to the cache's key population.

A flame graph attributes memory usage to the code responsible for allocation, not to the structure that ultimately holds the memory. In this codebase, allocation occurs during fetch and deserialisation, and nodes are subsequently managed by the cache through canonicalize. Therefore, the data identifies a likely source of memory usage, but confirming this would require further experimental validation.

Cache experiments

At this stage a likely cause had been identified. Over several months, a series of experiments was conducted to make the cache smaller, cheaper, or better behaved. Each experiment tested a specific theory regarding memory usage.

Theory: some caches may be unnecessary. The initial approach was to disable the TaggedCache and observe the impact (#5439, with initial commits titled "TEST: Disable tagged cache to measure performance"). Disabling the TaggedCache revealed its role as both a cache and the canonicalization point, as well as its function in warming the new database file during online-delete rotation. Complete removal was not feasible. The cache could only be made smaller or shorter-lived, and because rotation warms the new database file from the cache's keys, shrinking it trades memory for post-rotation latency. The same pull request then focused on the NodeObject cache in DatabaseNodeImp, which stored recently read database records in memory despite the underlying database engine and kernel page cache already retaining these records. Removing this cache reduced memory usage by 3–5% with a minor increase in CPU use.

Theory: the cache structure leads to memory waste on large hardware. The TaggedCache is partitioned per hardware thread, causing fixed overhead to scale with the number of cores rather than the workload. Reducing the partition count by half (#5442) resulted in a 5–6% memory reduction.

Theory: cache entries persist longer than necessary. The sweep evicts entries that are unreferenced or expired. To test this, a probe build (#5910) set all entries to expire immediately, converting each sweep into a full eviction. This approach reduced memory usage by 8–10%.

Theory: some structures are never cleaned. The LedgerHistory::ledgersByIndex_ map, which associates ledger sequence numbers with hashes, was not subject to sweeping and accumulated entries indefinitely. This resulted in a small but unbounded growth, a logical leak: entries were added for every validated ledger and never removed (#5461). It was the only such leak the investigation found. The fix in #5874 bounds the map on nodes with online delete enabled. Full-history nodes, which keep every ledger, still accumulate about 1.4 MB per day. The growth was measured in kilobytes per hour while the overall issue was measured in gigabytes, so it did not account for the observed memory growth.

Individually, each change showed progress. However, the combined effect was less than expected. While the sum of individual improvements suggested a 20% reduction, the cumulative measurement in a single build was only 5–7%. This discrepancy indicated either unreliable measurements or a lower bound imposed by factors outside the application. Both explanations turned out to be true, and the second was the larger problem. The experiments remain relevant, each demonstrating measurable effects with tools that have since been improved. Further testing in a rebuilt lab environment is planned, but the focus has shifted to addressing the underlying constraint.

Of the four changes, the NodeObject cache removal (#5439) was merged and later reverted as a precaution, without any problem having been attributed to it, the partition-count reduction (#5442) remains open, the sweep-everything build (#5910) was a probe never intended to ship, and the ledgersByIndex_ fix (#5874) was merged in October 2026.

Released memory is not reclaimed

In October 2025, the investigation moved from periodic snapshots to continuous monitoring. The memleak-based sampler measured TaggedCache's outstanding bytes, everything its stacks had allocated and not yet freed, every few tens of seconds during hour-long runs. In parallel, WSS tracked RSS and Referenced.

The results were clear. The outstanding allocations from the cache subsystem consistently returned to near zero on each sweep cycle, which occurred approximately every two minutes. The sweep-everything probe from #5910, which evicts the entire cache at each sweep, amplified this effect. Allocation and deallocation rates matched in real time on the unmodified build. Despite this, RSS and Referenced memory continued to increase.

TaggedCache outstanding bytes vs RSS/Referenced, 72 minutes, develop build

Seventy-two minutes on the unmodified develop build, connected to Mainnet, 21 October 2025. Top: bytes allocated by TaggedCache-machinery stacks and not yet freed — the memleak sampler, ~25 s snapshots. memleak reports the ten largest outstanding stacks per snapshot, so TaggedCache absent from the top ten plots as zero. Peak 64 MiB, median 2 MiB, every excursion collapsing within minutes (sweep interval ≈ 121 s). Bottom: the same process on the same clock — RSS climbs 13.7 → 15.5 GiB (+1.7 GiB, ~1.5 GiB/h) while Referenced converges toward it (wss.pl, cumulative mode). That rate is faster than September's 0.74 GiB/h — it varied with the run and the node's age — but the direction never did. One of four captures that day across both builds — the cache's outstanding bytes stayed in the MiB range in every one of them. This run is shown because it has a full RSS/Referenced companion on the same clock.

This combination of results rules out an application-level cause for the growth, including the cache. The cache's outstanding allocations are measured in MiB, while the observed memory growth is in GiB. Across all stacks reported by memleak, the median outstanding memory was 0.02 GiB, increasing by approximately 0.1 GiB during the run, compared to 1.7 GiB of RSS growth. Anything accumulating toward that figure would have appeared in the top ten stacks. Memory that the application has freed but the kernel still counts as resident, and that is still being touched, is not a leak and does not indicate cache malfunction. This behaviour is the allocator's.

When xrpld calls free, glibc's malloc typically retains the pages on its internal free lists, anticipating future reuse by the application. The kernel cannot request memory back from the allocator. Under memory pressure it can drop page cache or swap pages out, but it cannot instruct malloc to release its holdings. glibc only returns free space at the end of each heap when the unused tail exceeds a threshold. Freed memory blocks that are interleaved with live allocations, as is common with caches containing objects of mixed lifetimes, remain stranded within the heap. This is fragmentation. The allocation pattern of an XRPL node, which involves bursts of millions of small, short-lived tree nodes at sweep and rotation boundaries, and the use of per-thread arenas (each busy thread receives its own heap region), leads to fragmentation in every arena. This accumulates to gigabytes of memory that are free from the application's perspective but remain resident according to system metrics.

The key distinction is between memory released by the application and memory reclaimed by the allocator. The application can release memory, but only the allocator can reclaim it. There was no mechanism in the node's code path to connect these actions.

The investigation focused on identifying a memory leak. No leak of any consequence was found. The allocator was retaining memory as part of a general-purpose strategy that is poorly suited to this workload. The application can precisely identify when memory is no longer needed, but lacks a mechanism to inform the allocator.

There is a solution available. glibc provides malloc_trim, which can be invoked by xrpld at an appropriate point in its operation. The implementation details, performance impact, and effects on memory usage in Mainnet validators will be discussed in the next post.

The diagnostic toolset fulfilled its purpose by identifying the root cause: the allocator handled xrpld's allocation pattern poorly, and neither side had a way to tell the other what it knew.

Acknowledgements

This investigation began with reports from operators running the software on Mainnet, who measured the growth on their own nodes, compared notes on how to measure it, and kept asking the questions that turned out to matter. Thanks are due to them, to those who reviewed the change in the open-source repository, and to those who reviewed these two posts.