PostgreSQL document compression: TOAST thresholds, storage policies, and page size

A common way to store documents in PostgreSQL is a table with an ID and a JSONB column. The open-source DocumentDB extension instead stores BSON, the binary document format used by MongoDB, to provide more data types and MongoDB compatibility.

Both approaches commonly combine two patterns that do not perfectly fit what SQL databases were designed for:

• Rows with a single variable-length column, whose values can range from a few dozen bytes to many megabytes.
• Repeated data or metadata, such as similar field names and arrays of objects with the same schema, like time-series buckets.

MongoDB's storage engine, WiredTiger, packs many documents into B-tree leaf pages. The default maximum page sizes — 32 KB for leaf pages and 4 KB for internal pages — are limits on the uncompressed page image fed to the block compressor, not on individual documents: MongoDB configures leaf_value_max=64MB, so a document larger than 32 KB simply produces a larger leaf page rather than overflow storage. Snappy compresses each page image for ordinary collections (time-series collections use Zstd, which ignores the global compressor setting), and the compressed result is written in multiples of the 4 KB allocation_size. Because compression runs over the whole page image, several small documents — including their repeated field names — contribute redundant byte sequences to the same compression input.

A JSON or BSON document can contain substantial byte-sequence redundancy: repeated field names, common name prefixes, and repeated values or structures. LZ77-family compressors such as Snappy and LZ4 can replace matching sequences with backreferences to earlier bytes. They don't need to understand JSON, and matches aren't restricted to the beginning of a document.

Binary does not mean compressed: BSON includes field names, and repeated embedded objects repeat those names. JSONB can contain similar redundancy. PostgreSQL can compress those sequences in JSON, JSONB, or a BSON extension type. However, the decision to attempt compression is not driven by document redundancy. It belongs to TOAST's generic heap-row storage policy: reduce a wide row, trying column compression before storing permitted attributes externally in chunks.

This distinction matters for small documents. It also makes increasing the page size potentially counterproductive: the same document can stop being compressed on a larger-page build.

I will reproduce both effects, then compare the SQL controls with the hardcoded values. The examples use PostgreSQL 19 and core JSONB, so they can run without an extension or a custom patch. The distinction between whole-row activation and individual-column compression also applies to the TOAST-backed BSON values stored by DocumentDB.

What is compressed, and what is measured?

TOAST measures the whole tuple (row) to decide whether to start. However, compression acts on individual eligible column values, not on the entire row or all the documents in a page.

PostgreSQL's ordinary table storage uses fixed-size pages, 8 KB by default. A row cannot span those pages. Large column values can instead be moved to an associated TOAST table, and the main row then contains a pointer to access the chunks in the TOAST table by index scan.

JSONB still contains compressible byte sequences. It is a binary representation, not an already-compressed JSON string. Within one object, it normalizes key order and duplicate keys, but an array of similarly structured objects can repeat the same field names. No shared key dictionary exists across those objects or across independently stored JSONB rows.

The distinction between repeated names within a document and repeated names across documents is important. PostgreSQL compresses one column value at a time. PostgreSQL calls each value a datum in its source code. That per-datum compression can use the former repetition, not the latter.

MongoDB's WiredTiger storage engine uses Snappy block compression on disk by default for ordinary collections. Small documents can share a page, so one block-compression input can include repeated names and structures from several documents. This is not a separate compression call per small document. Other codecs can be configured, and time-series collections have different defaults. This is a difference in compression scope, not a capability reserved to Snappy.

This post measures stored datum representations, not table allocation or query performance. I use:

• pg_column_size(doc::text::jsonb) to reconstruct an uncompressed logical JSONB datum for these ordinary JSON inputs
• pg_column_compression(doc) to report the actual accepted method
• pg_column_size(doc) for its stored representation
• pg_column_toast_chunk_id(doc) IS NOT NULL to identify external values

The last function is available in PostgreSQL 19. For an external value, pg_column_size is not the inline pointer size. The on-disk pointer in the main row occupies 18 bytes. The external size excludes the outer four-byte varlena length word, which explains some four-byte differences below without attributing them to the codec.

Reproduce the environment and output

All SQL needed for the experiments is included below. Run the statements in order in a fresh test database. For each page-size comparison, start a separate PostgreSQL instance.

I use aligned ASCII output, with NULL displayed as (null). To reproduce that format, start an existing server's session with these options:

psql -X -P pager=off -P 'null=(null)' -P format=aligned \ -P linestyle=ascii -P border=1 -U postgres -d postgres 

These are the initial settings on the source-built 8 KB server:

postgres=# SHOW server_version; server_version ---------------- 19rc1 (1 row) postgres=# SHOW block_size; block_size ------------ 8192 (1 row) postgres=# SHOW default_toast_compression; default_toast_compression --------------------------- lz4 (1 row) 

The selected codec is explicit in every test. PostgreSQL 14-18's upstream default is pglz, but an LZ4-enabled PostgreSQL 19 RC1 build defaults to LZ4. SET COMPRESSION has been available since PostgreSQL 14.

LZ4 is a sensible starting choice when compression/decompression CPU matters, but the workload still determines the trade-off. It does not always produce the smallest result. DocumentDB's current setup/tuning configuration selects default_toast_compression = 'lz4' when supported. This is a launcher-chosen PostgreSQL setting, not a property of BSON, nor a guarantee that every installation is configured that way. Check the live server and column settings rather than assuming them from a product name.

A compressible document below the activation threshold

I construct a JSONB array containing eight objects with repeated temperature and humidity names. Both rows receive the same document. The only difference is an unrelated column stored PLAIN, so it cannot be compressed or moved externally.

postgres=# CREATE TABLE small_documents( id integer, doc jsonb, other text ); CREATE TABLE postgres=# ALTER TABLE small_documents ALTER COLUMN doc SET COMPRESSION lz4 ; ALTER TABLE postgres=# ALTER TABLE small_documents ALTER COLUMN other SET STORAGE PLAIN ; ALTER TABLE postgres=# INSERT INTO small_documents SELECT id, jsonb_build_object('samples', ( SELECT jsonb_agg(jsonb_build_object('temperature',20,'humidity',60)) FROM generate_series(1,8) )), repeat('q', CASE id WHEN 1 THEN 0 ELSE 1600 END) FROM generate_series(1,2) AS g(id) ; INSERT 0 2 postgres=# SELECT id, pg_column_size(doc::text::jsonb) AS logical_bytes, octet_length(other) AS other_bytes, pg_column_compression(doc) AS compression, pg_column_size(doc) AS stored_bytes, pg_column_toast_chunk_id(doc) IS NOT NULL AS external FROM small_documents ORDER BY id ; id | logical_bytes | other_bytes | compression | stored_bytes | external ----+---------------+-------------+-------------+--------------+---------- 1 | 508 | 0 | (null) | 508 | f 2 | 508 | 1600 | lz4 | 109 | f (2 rows) 

The first 508-byte JSONB datum remains uncompressed. The second compresses to 109 bytes, even though its uncompressed document bytes are identical. The additional 1,600-byte column made the tuple large enough to enter TOAST processing.

This proves both points:

1. Repeated field names and values in JSONB are compressible.
2. PostgreSQL does not independently ask whether that small JSONB datum would benefit from compression. The enclosing tuple controls activation.

I used an unrelated column for academic purposes. Of course, I'm not suggesting that adding more data to trigger compression is a good idea.

The hardcoded gate is not the per-table target

In the PostgreSQL source, the ordinary values are derived as follows:

#define TOAST_TUPLES_PER_PAGE 4 #define TOAST_TUPLE_THRESHOLD MaximumBytesPerTuple(TOAST_TUPLES_PER_PAGE) #define TOAST_TUPLE_TARGET TOAST_TUPLE_THRESHOLD 

MaximumBytesPerTuple(4) accounts for page headers, line pointers, and alignment. It is not exactly block_size / 4. Here are the values for the default 8 KB page size and for PostgreSQL builds with 16 KB or 32 KB pages:

These are the values for the tested eight-byte-aligned builds. The chunk capacities were also observed from actual external data, not inferred from page size alone.

Both heap_prepare_insert and heap_update check the fixed threshold before invoking the toaster for fresh inline tuples. Inside the toaster, toast_tuple_target provides the reduction target, after subtracting tuple header overhead.

It’s not simply "compress when the row cannot fit on a page.” A tuple just over 2 KB can still fit on an 8 KB page. By default, wide rows are reduced earlier, with the goal of keeping their size consistent with roughly four tuples per page. This is a row-density and oversized-attribute policy, rather than a separately tunable document-compression policy.

These are fresh values with one eligible document column. If a row has several eligible columns, the toaster works in phases rather than compressing the entire row as one object.

Change the target without changing activation

The following input view constructs fresh JSONB values from repetitive text. It deliberately samples several payload sizes. It is not intended to represent a natural document distribution.

A view is used rather than copying stored compressed values from another table. Reusing an existing compressed datum would confound a test of whether the destination attempts compression.

postgres=# CREATE TEMP VIEW input_documents AS SELECT n, jsonb_build_object('v', repeat('x', 1024 * (1 << n))) AS doc FROM (VALUES (0),(1),(2),(6),(9),(10),(11)) AS sizes(n) ; CREATE VIEW postgres=# CREATE TABLE extended_default( n integer PRIMARY KEY, doc jsonb ); CREATE TABLE postgres=# CREATE TABLE extended_200( n integer PRIMARY KEY, doc jsonb ) WITH (toast_tuple_target=200) ; CREATE TABLE postgres=# CREATE TABLE extended_4k( n integer PRIMARY KEY, doc jsonb ) WITH (toast_tuple_target=4096) ; CREATE TABLE postgres=# ALTER TABLE extended_default ALTER COLUMN doc SET COMPRESSION lz4 ; ALTER TABLE postgres=# ALTER TABLE extended_200 ALTER COLUMN doc SET COMPRESSION lz4 ; ALTER TABLE postgres=# ALTER TABLE extended_4k ALTER COLUMN doc SET COMPRESSION lz4 ; ALTER TABLE postgres=# INSERT INTO extended_default SELECT * FROM input_documents ORDER BY n ; INSERT 0 7 postgres=# INSERT INTO extended_200 SELECT * FROM input_documents ORDER BY n ; INSERT 0 7 postgres=# INSERT INTO extended_4k SELECT * FROM input_documents ORDER BY n ; INSERT 0 7 postgres=# SELECT policy, 1 << n AS payload_kib, pg_column_compression(doc) AS compression, pg_column_size(doc) AS stored_bytes, pg_column_toast_chunk_id(doc) IS NOT NULL AS external FROM ( SELECT 1 AS position, 'default' AS policy, * FROM extended_default UNION ALL SELECT 2,'target_200',* FROM extended_200 UNION ALL SELECT 3,'target_4096',* FROM extended_4k ) AS cases ORDER BY position,n ; policy | payload_kib | compression | stored_bytes | external -------------+-------------+-------------+--------------+---------- default | 1 | (null) | 1041 | f default | 2 | lz4 | 39 | f default | 4 | lz4 | 47 | f default | 64 | lz4 | 288 | f default | 512 | lz4 | 2083 | t default | 1024 | lz4 | 4139 | t default | 2048 | lz4 | 8252 | t target_200 | 1 | (null) | 1041 | f target_200 | 2 | lz4 | 39 | f target_200 | 4 | lz4 | 47 | f target_200 | 64 | lz4 | 284 | t target_200 | 512 | lz4 | 2083 | t target_200 | 1024 | lz4 | 4139 | t target_200 | 2048 | lz4 | 8252 | t target_4096 | 1 | (null) | 1041 | f target_4096 | 2 | (null) | 2065 | f target_4096 | 4 | lz4 | 47 | f target_4096 | 64 | lz4 | 288 | f target_4096 | 512 | lz4 | 2087 | f target_4096 | 1024 | lz4 | 4139 | t target_4096 | 2048 | lz4 | 8252 | t (21 rows) 

The default JSONB storage strategy is EXTENDED, which permits both compression and external storage.

Three observations distinguish the decisions:

• Even with a 200-byte target, the 1 KB-payload document still remains uncompressed and inline at 1,041 bytes. Its tuple did not enter TOAST because it is smaller than the hardcoded MaximumBytesPerTuple(4).
• Once TOAST is active, lowering the target externalizes values sooner: the 64 KB payload is inline with the default target but external with the 200-byte target.
• With a 4 KB target, the 2 KB-payload document remains uncompressed at 2,065 bytes. Its tuple enters TOAST, but already meets the larger target. The same document compressed to 39 bytes in the other two tables.

Raising the target is therefore not an independent "retain more compressed documents inline" control. It can also prevent a compression attempt.

The default and 200-byte cases report 288 and 284 bytes respectively for the same compressed 64 KB-payload document. This is the inline/external varlena accounting difference, not a better compression result.

Storage strategies: what SQL can control

There are four column storage strategies:

MAIN does not guarantee inline storage. MAIN attributes are also skipped during the first compression pass: they get their own compression round only after EXTENDED/EXTERNAL attributes have been compressed and externalized. They are likewise skipped by the out-of-line loops until the last-resort pass, which uses TOAST_TUPLE_TARGET_MAIN (MaximumBytesPerTuple(1) ≈ 8,160 bytes on an 8 KB build). EXTERNAL does not mean "externalize every document,” and an explicit LZ4 method cannot override its prohibition on compression.

Here EXTERNAL and MAIN use the same 4 KB target, making the comparison independent of an accidental target difference:

postgres=# CREATE TABLE external_4k( n integer PRIMARY KEY, doc jsonb ) WITH (toast_tuple_target=4096) ; CREATE TABLE postgres=# CREATE TABLE main_4k( n integer PRIMARY KEY, doc jsonb ) WITH (toast_tuple_target=4096) ; CREATE TABLE postgres=# CREATE TABLE plain_case( n integer PRIMARY KEY, doc jsonb ); CREATE TABLE postgres=# ALTER TABLE external_4k ALTER COLUMN doc SET COMPRESSION lz4 ; ALTER TABLE postgres=# ALTER TABLE main_4k ALTER COLUMN doc SET COMPRESSION lz4 ; ALTER TABLE postgres=# ALTER TABLE plain_case ALTER COLUMN doc SET COMPRESSION lz4 ; ALTER TABLE postgres=# ALTER TABLE external_4k ALTER COLUMN doc SET STORAGE EXTERNAL ; ALTER TABLE postgres=# ALTER TABLE main_4k ALTER COLUMN doc SET STORAGE MAIN ; ALTER TABLE postgres=# ALTER TABLE plain_case ALTER COLUMN doc SET STORAGE PLAIN ; ALTER TABLE postgres=# INSERT INTO external_4k SELECT * FROM input_documents ORDER BY n ; INSERT 0 7 postgres=# INSERT INTO main_4k SELECT * FROM input_documents ORDER BY n ; INSERT 0 7 postgres=# INSERT INTO plain_case SELECT * FROM input_documents WHERE n<=2 ORDER BY n ; INSERT 0 3 postgres=# SELECT policy, 1 << n AS payload_kib, pg_column_compression(doc) AS compression, pg_column_size(doc) AS stored_bytes, pg_column_toast_chunk_id(doc) IS NOT NULL AS external FROM ( SELECT 1 AS position,'EXTERNAL' AS policy,* FROM external_4k UNION ALL SELECT 2,'MAIN',* FROM main_4k UNION ALL SELECT 3,'PLAIN',* FROM plain_case ) AS cases ORDER BY position,n ; policy | payload_kib | compression | stored_bytes | external ----------+-------------+-------------+--------------+---------- EXTERNAL | 1 | (null) | 1041 | f EXTERNAL | 2 | (null) | 2065 | f EXTERNAL | 4 | (null) | 4109 | t EXTERNAL | 64 | (null) | 65549 | t EXTERNAL | 512 | (null) | 524301 | t EXTERNAL | 1024 | (null) | 1048589 | t EXTERNAL | 2048 | (null) | 2097165 | t MAIN | 1 | (null) | 1041 | f MAIN | 2 | (null) | 2065 | f MAIN | 4 | lz4 | 47 | f MAIN | 64 | lz4 | 288 | f MAIN | 512 | lz4 | 2087 | f MAIN | 1024 | lz4 | 4143 | f MAIN | 2048 | lz4 | 8252 | t PLAIN | 1 | (null) | 1041 | f PLAIN | 2 | (null) | 2065 | f PLAIN | 4 | (null) | 4113 | f (17 rows) 

For EXTERNAL, the 2 KB payload is still inline, but the 4 KB payload is external without compression.

For MAIN, the 1 MB payload compresses to 4,143 inline bytes. It can stay inline because MAIN postpones externalization to its last-resort target: the maximum tuple size that fits on one page. The larger 2 MB-payload document still becomes external after compression.

For PLAIN, no value compresses or becomes external. A row that cannot fit on the heap page fails. Run this separate statement expecting an error:

postgres=# INSERT INTO plain_case VALUES (3,jsonb_build_object('v',repeat('x',8192))) ; ERROR: 54000: row is too big: size 8240, maximum size 8160 LOCATION: RelationGetBufferForTuple, hio.c:531 

The 8,160...