Fix OOM Kills and MEMORY_LIMIT_EXCEEDED on ClickHouse® Servers

Introduction

ClickHouse® servers can experience two types of memory-related failures.

In the first, the Linux Out-Of-Memory (OOM) killer terminates clickhouse-server. The service restarts, and all running queries fail.

In the second, multiple queries fail with MEMORY_LIMIT_EXCEEDED (code 241). The error message contains (total), indicating that the entire server has reached its memory limit rather than an individual query exceeding its own limit.

Both failures have the same underlying cause: the server is attempting to use more memory than the host can provide.

In this guide, we'll explain how ClickHouse manages server memory, how to identify the source of memory failures, and how to configure appropriate limits to protect your server.

1. Why Memory Failures Happen

ClickHouse maintains a hard memory limit for the entire server. When a query attempts an allocation that would exceed this limit, the query is stopped.

Two steps determine the server's memory limit.

Step 1: Establishing the Memory Ceiling

At startup, ClickHouse calculates its memory ceiling using the available memory and the setting max_server_memory_usage_to_ram_ratio.

The default value is 0.9, meaning ClickHouse can use up to 90% of the available memory for its server memory limit.

The setting max_server_memory_usage defaults to 0, which means no fixed limit is configured. If you specify a fixed value, ClickHouse uses the lower of that value and the calculated ceiling.

For example, on a host with 13.55 GiB of RAM, setting a fixed limit of 15.63 GiB resulted in the following log message:

Lowered setting 'max_server_memory_usage' to 12.20 GiB because the system has too little memory. The new value was calculated as 13.55 GiB available memory * 0.90 max_server_memory_usage_to_ram_ratio 

ClickHouse automatically lowered the configured limit because the host could not support the requested amount.

Step 2: Understanding the Runtime Memory Limit

ClickHouse also uses a background thread called the memory worker to recalculate the hard memory limit while the server is running.

The runtime limit is calculated using:

(resident memory + available system memory) * ratio 

The result cannot exceed the previously established ceiling.

When other processes consume memory, the available system memory decreases, and the runtime limit can fall accordingly.

For example:

MemoryWorker: Adjusting total memory hard limit from 5.62 GiB to 5.69 GiB (resident: 364.99 MiB, available: 5.97 GiB, ceiling: 12.20 GiB, ratio: 0.9) 

The calculation is approximately:

(0.36 GiB + 5.97 GiB) × 0.9 = 5.69 GiB 

Although the ceiling remained at 12.20 GiB, the live limit stayed around 6 GiB because other processes were consuming some of the host's memory.

Why the Safety Margin Matters

The ratio provides a safety margin because ClickHouse's memory tracker does not account for every memory allocation.

In one test, MemoryResident was 940 MiB while TrackedMemory was only 365 MiB. The remaining memory was consumed by the kernel, other services, and allocator overhead.

A sudden memory spike can consume the available margin between memory-worker checks, potentially triggering the Linux OOM killer.

What Happens in Containers?

ClickHouse can read memory limits imposed through cgroups.

For example, after setting MemoryMax=8G with systemd and restarting the server, the log showed:

Changed setting 'max_server_memory_usage' to 7.20 GiB (8.00 GiB available memory * 0.90 max_server_memory_usage_to_ram_ratio) 

Without a container memory limit, ClickHouse may see the memory available on the entire host rather than the amount intended for the service.

The setting memory_worker_use_cgroup controls the use of cgroup memory information for runtime memory-limit calculations. However, a test on a host with similar free memory inside and outside the limit does not, by itself, prove the behavior.

Version note: In ClickHouse 26.3.9, memory_worker_dynamic_hard_limit does not exist. The limit remains at the ceiling, making the memory ratio an important protection mechanism.

2. How to Confirm the Problem

Before changing settings, determine whether the operating system killed the process or ClickHouse stopped a query because of its own memory limit.

Step 1: Check for OOM Kills

Run:

sudo dmesg -T | grep -i -E "out of memory|killed process" 

You can also inspect kernel logs from the last 24 hours:

sudo journalctl -k --since "24 hours ago" | grep -i -E "out of memory|killed process" 

Check when the ClickHouse service last started:

systemctl show clickhouse-server -p ActiveEnterTimestamp 

If the service started recently and the kernel logs show that clickhouse-serv was killed, the operating system likely terminated the process due to memory pressure.

The kernel truncates process names to 15 characters, which explains why the process name may appear as clickhouse-serv.

Step 2: Find Server-Level Memory Errors

A ClickHouse server-level memory failure typically appears as MEMORY_LIMIT_EXCEEDED with error code 241 and (total) in the message.

For example, this error was observed on ClickHouse 26.8.36:

Code: 241. DB::Exception: (total) memory limit exceeded: would use 6.28 GiB (attempt to allocate chunk of 4.00 GiB), current RSS: 2.23 GiB, maximum: 6.08 GiB. OvercommitTracker decision: Query was selected to stop by OvercommitTracker. Untracked memory across all threads: -6.82 MiB: While executing AggregatingTransform. (MEMORY_LIMIT_EXCEEDED) 

The maximum value represents the live server memory limit. It may be lower than the configured ceiling.

To find recent server-level memory errors, query system.query_log:

SELECT event_time, query_id, substring(exception, 1, 250) AS exception_start FROM system.query_log WHERE type = 'ExceptionWhileProcessing' AND exception_code = 241 AND exception LIKE '%(total)%' AND event_time > now() - INTERVAL 1 DAY ORDER BY event_time DESC LIMIT 20; 

The system.errors table contains error counts accumulated since the server started and is reset on restart. The query log provides historical records of individual failures.

Step 3: Compare Process Memory with Host Memory

Run the following query:

SELECT metric, formatReadableSize(value) AS size FROM system.asynchronous_metrics WHERE metric IN ( 'MemoryResident', 'TrackedMemory', 'OSMemoryTotal', 'OSMemoryAvailable' ) ORDER BY metric; 

These metrics help you compare the memory used by ClickHouse with the total and available memory on the host.

Step 4: Read the Live Memory Limit

Use system.server_settings to inspect the current server memory limit:

SELECT formatReadableSize(toUInt64(value)) AS live_limit FROM system.server_settings WHERE name = 'max_server_memory_usage'; 

This shows the effective limit, which may differ from the value in your configuration file.

For more information about runtime adjustments, inspect the server logs:

sudo grep -a "Adjusting total memory hard limit" /var/log/clickhouse-server/clickhouse-server.log | tail -5 

These messages are available at Trace log level.

3. Understanding the Relevant Settings

The following defaults are based on ClickHouse 26.8.36. Verify them on your own server before making changes.

Run:

SELECT name, value, default, changed, changeable_without_restart FROM system.server_settings WHERE name LIKE 'max_server_memory%' OR name LIKE 'memory_worker%'; 

The important settings are:

The changeable_without_restart column may show No for the ratio setting. In the test described here, reloading the configuration still applied the change, and the logs showed a new Changed setting message.

If the expected log message does not appear after reloading the configuration, restart the server.

4. How to Fix Memory Failures

The recommended approach is to reserve memory for the operating system and configure explicit memory limits for containers.

Step 1: Set a Lower Memory Ratio

Create a configuration file at:

/etc/clickhouse-server/config.d/memory.xml 

Add the following configuration:

<clickhouse> <max_server_memory_usage_to_ram_ratio>0.8</max_server_memory_usage_to_ram_ratio> </clickhouse> 

A ratio of 0.8 allows ClickHouse to use up to 80% of the available memory for its server memory limit.

For example, on a host with 64 GiB of memory, the ceiling would be 51.2 GiB, leaving 12.8 GiB for the operating system and other processes.

Consider these starting points:

• 0.8: A reasonable starting point for many hosts.
• 0.7: Consider this when the host runs other services.
• 0.85: Consider this only on a dedicated host with carefully monitored memory usage.

These values are starting points, not universal recommendations. The appropriate ratio depends on the host, workload, and memory consumed by other processes.

Reload the configuration:

SYSTEM RELOAD CONFIG; 

Then verify the updated setting in the logs:

sudo grep -a "max_server_memory_usage" /var/log/clickhouse-server/clickhouse-server.log | tail -3 

You should see a Changed setting message confirming the new value.

For example, a test using a ratio of 0.15 produced:

Changed setting 'max_server_memory_usage' to 2.03 GiB (13.55 GiB available memory * 0.15 max_server_memory_usage_to_ram_ratio) 

Configuration tip: Write the new file in a separate directory first, then copy it into config.d/. This reduces the chance of ClickHouse loading a partially written configuration file.

Step 2: Configure a Fixed Memory Limit

If you know how much memory ClickHouse should use on a shared host, you can set an explicit limit alongside the ratio.

Update memory.xml:

<clickhouse> <max_server_memory_usage_to_ram_ratio>0.8</max_server_memory_usage_to_ram_ratio> <max_server_memory_usage>53687091200</max_server_memory_usage> </clickhouse> 

The value 53687091200 represents 50 GiB in bytes.

A fixed limit makes the memory budget easier to understand and manage. ClickHouse uses the lowest applicable value among the fixed limit, the calculated ceiling, and the runtime limit.

Reload the configuration:

SYSTEM RELOAD CONFIG; 

In the test described here, changing the configured value from 3 GiB to 2 GiB through a configuration reload caused the next memory error to report:

maximum: 2.00 GiB 

Verify the effective limit in the logs and through system.server_settings.

Step 3: Set a Container Memory Limit

A container memory limit prevents ClickHouse from consuming the entire host's memory.

Docker example:

docker run -d --name clickhouse --memory=32g clickhouse/clickhouse-server 

Kubernetes example:

resources: requests: memory: 32Gi limits: memory: 32Gi 

Setting the Kubernetes memory request equal to the limit reserves the same memory amount for scheduling purposes and helps avoid unexpected memory-pressure evictions caused by overcommitting the requested amount.

With a 32 GiB container limit and a ratio of 0.8, the calculated ceiling is 25.6 GiB.

Systemd's MemoryMax provides a similar cgroup memory limit.

Remember that the container limit and ClickHouse's internal memory limit serve different purposes: the container constrains the process at the operating-system level, while ClickHouse's settings help it stop queries before reaching that boundary.

5. How to Verify the Fix

After changing the configuration, check the effective settings and confirm that the logs show the expected memory limit.

On a test server, you can trigger a memory-limit error using:

SELECT groupArray(number) FROM numbers(100000000000) SETTINGS max_memory_usage = 0 FORMAT Null; 

Warning: Do not run this test on a production server. It is intended for controlled testing only.

The query should fail with code 241. The maximum value in the error should match the configured effective limit or be lower if the host has insufficient available memory.

For production, monitor the server for seven days after applying the changes.

The fix is working as intended when:

1. Kernel logs show no new OOM kills.
2. The query log contains no new unexpected (total) memory-limit errors.
3. MemoryResident remains below OSMemoryTotal with a safe memory margin.

Continue monitoring workloads that generate large aggregations, memory-intensive queries, merges, and mutations.

6. Risks and Trade-Offs

Lower Limits Can Cause Queries to Fail Earlier

Reducing the memory limit means ClickHouse may reject queries that previously succeeded.

This is often the correct trade-off: a failed query is preferable to a server process being killed and disrupting all running queries.

Identify and tune the heaviest queries rather than simply increasing the server limit.

The Live Limit Can Change

On a shared host, the effective memory limit can decrease when other processes consume more memory.

As a result, queries may fail even when the configured settings have not changed. Investigate other processes and host-level memory usage before increasing ClickHouse's limits.

Merges and Mutations Have Their Own Memory Controls

Merges and mutations have a separate setting:

merges_mutations_memory_usage_to_ram_ratio 

Its default is 0.5. Review it when diagnosing memory pressure related to background operations.

Caches Also Consume Memory

Caches contribute to memory consumption. Avoid increasing cache sizes when lowering the server's memory limit unless you have measured the impact and confirmed that sufficient memory remains available.

Be Careful When Using Swap

The setting description notes that low-RAM hosts with swap may require a ratio above 1 in certain situations.

This should only be done deliberately. Allowing ClickHouse to use swap before reaching its configured limit may delay an OOM event, but heavy swapping can make queries extremely slow and postpone rather than prevent a more severe failure.

For ClickHouse hosts, consider disabling swap or configuring a low vm.swappiness value according to your operational requirements.

7. Summary

When troubleshooting ClickHouse memory failures:

• Check dmesg and the service start time to determine whether the operating system killed the process.
• Look for MEMORY_LIMIT_EXCEEDED code 241 errors containing (total) to identify server-level memory failures.
• Read the maximum value in the error message to understand the effective live limit.
• Remember that the live limit is constrained by the fixed limit, the ratio-based ceiling, and the runtime limit calculated by the memory worker.
• Configure a suitable max_server_memory_usage_to_ram_ratio in config.d/ and reload the configuration.
• Set an explicit container memory limit when running ClickHouse in Docker, Kubernetes, or another cgroup-managed environment.
• Monitor kernel logs, query errors, and memory metrics after applying the fix.

The goal is not to let ClickHouse consume every available byte of memory. It is to establish a predictable memory budget that protects the server while allowing normal workloads to run reliably.

References

• ClickHouse documentation: max_server_memory_usage and max_server_memory_usage_to_ram_ratio.