1.5 GiB in, 2.1 GiB out: OpenSSL's base64 overflows an int

One byte past 1,610,612,733 and EVP_EncodeBlock() writes 2,147,483,648 bytes into a buffer your own size math said was −2,147,483,647 bytes long. I ran it against the OpenSSL 3.0.13 on my machine: it returned −2,147,483,648, and AddressSanitizer caught the verbatim encoder writing past a 1 MiB buffer.

The bug is CVE-2026-121481, scored 9.8, disclosed this week in OpenSSL pull request #33652. EVP_EncodeBlock() takes an int dlen and returns an int — the length of the encoded output. Base64 expands 3 bytes to 4, so the output is 4 * ceil(dlen / 3) bytes. That only fits in a signed 32-bit int while dlen <= (INT_MAX / 4) * 3, which is 1,610,612,733 bytes — about 1.5 GiB. One byte more and the length wraps. The function never checks. It just keeps writing.

The threshold, computed

INT_MAX = 2,147,483,647 threshold (INT_MAX/4)*3 = 1,610,612,733 (1.500 GiB) dlen=1,610,612,733 -> int formula = 2,147,483,645, real bytes = 2,147,483,645, overflow = no dlen=1,610,612,734 -> int formula = -2,147,483,647, real bytes = 2,147,483,649, overflow = YES 

The "int formula" column is what a caller gets from the documented sizing rule, 4 * ((dlen + 2) / 3) + 1, computed in 32-bit int. At the threshold it is fine. One byte past it, the same expression wraps to a negative number. The encoder, meanwhile, really writes 2,147,483,649 bytes. The gap between those two numbers is the overflow.

Experiment 1: AddressSanitizer on the verbatim encoder

I took evp_encodeblock_int() verbatim from OpenSSL 3.0's crypto/evp/encode.c — the scalar loop, unchanged — compiled it with -fsanitize=address, and called it with dlen = 1,610,612,734 and a 1 MiB output buffer:

dlen=1610612734, out buffer=1 MiB, calling encoder... ==11787==ERROR: AddressSanitizer: heap-buffer-overflow WRITE of size 1 #0 in main 0x70b2c18f6800 is located 0 bytes after 1048576-byte region [0x70b2c17f6800,0x70b2c18f6800) SUMMARY: AddressSanitizer: heap-buffer-overflow in main 

The loop is almost comically simple — for (i = dlen; i > 0; i -= 3), four bytes out per iteration, ret += 4:

for (i = dlen; i > 0; i -= 3) { if (i >= 3) { l = (((unsigned long)f[0]) << 16L) | (((unsigned long)f[1]) << 8L) | f[2]; *(t++) = conv_bin2ascii(l >> 18L, table); *(t++) = conv_bin2ascii(l >> 12L, table); *(t++) = conv_bin2ascii(l >> 6L, table); *(t++) = conv_bin2ascii(l, table); } else { /* ... padding path, same 4-byte write ... */ } ret += 4; f += 3; } *t = '\0'; return ret; 

Nothing in that loop knows or cares how big the destination is. ret wraps to a negative number around iteration 536 million, and the writes just continue. The only thing that stops it in my test is ASan's redzone. In a real program, the only thing that stops it is the end of the heap.

Experiment 2: the shipped library, 1.5 GiB for real

The ASan test used a copy of the function, so I also ran the real thing: the system libcrypto, OpenSSL 3.0.13, linked normally. Input was 1,610,612,734 bytes of A in a file on disk, memory-mapped — the encoder reads it sequentially, so the page cache reclaims behind it and resident memory stays small. Output went to a 2,147,483,649-byte sparse file, also memory-mapped. (My VM has 8 GiB of RAM with about 500 MB free; allocating 3.7 GiB of buffers was not an option, which is why both sides are files.)

OpenSSL 3.0.13 30 Jan 2024, dlen=1610612734, calling EVP_EncodeBlock... returned -2147483648 (expected 2147483648, wraps to -2147483648) out[0..3]=QUFB out[2147483644..2147483648]=QQ== out[2147483648]=0 

Three observations, all from the real library:

  • The return value is −2,147,483,648. The true output length is 2,147,483,648. The int return type cannot express it, so the function reports a negative length for a successful encode.
  • The output itself is correct base64: it starts QUFB and ends QQ== followed by the NUL terminator, exactly as 1,610,612,734 A bytes should encode. The scalar encoder does not corrupt data at this size — it just cannot tell you how much it wrote.
  • The encode took 20 seconds wall clock, 1.6 seconds of CPU. The rest was the kernel writing 2.1 GiB of output back to disk. The bug is not a performance problem; it is a bookkeeping problem with a 2 GiB blast radius.

So which callers actually overflow? The ones who do what the documentation tells them to. The man page says the destination must hold 4 * ((n + 2) / 3) + 1 bytes, and n is an int. Compute that in int and you get a negative size for any input over the threshold; pass the negative to malloc and you get NULL (or, with a cast, something worse); hand the result to EVP_EncodeBlock and it writes 2.1 GiB into it. The newer AVX2 path has the same shape — EVP_EncodeBlock returns (int)encode_base64_avx2(...), truncating a size_t result to int.

Notably, the streaming API already knew about this. EVP_EncodeUpdate() has an explicit guard:

if (total > INT_MAX) { /* Too much output data! */ *outl = 0; return 0; } 

The one-shot EVP_EncodeBlock() was the hole. Same library, same file, one function guarded and the other not.

The fix, and what it actually is

PR #33652, by nmouha, is open at the time of writing — not yet merged, and the formal advisory is still pending. The proposed fix adds two asserts to EVP_EncodeBlock():

OPENSSL_assert(dlen >= 0); OPENSSL_assert(dlen <= (INT_MAX / 4) * 3); 

plus a guard in EVP_DecodeBlock() (0 for empty input, −1 for negative length) and a corrected length check in http_client.c's base64encode(), where the old test was len > INT_MAX — reachable, per the backport notes, through a long s_client -proxy_pass password.

Two things about the fix are worth stating precisely, because both are easy to get wrong:

First, OPENSSL_assert is not compiled out in release builds. I checked the macro in include/openssl/crypto.h.in — it unconditionally calls OPENSSL_die() on failure. The NDEBUG that OpenSSL's Configure adds for release builds does not affect it. So the proposed fix turns a silent heap overflow into a loud, immediate abort. That is the right trade for a library function, and it is not the "asserts disappear in production" story.

Second, an abort is still not an error return. The PR's own comments point at follow-up issue #33812, which would move the check into the internal functions and return success/failure instead of dying. A caller handing EVP_EncodeBlock() a 2 GiB buffer today gets memory corruption; with this fix it gets a dead process. Better, and still not good API design — but you take the patch that stops the bleeding.

What to do about it

  • If you call EVP_EncodeBlock() or EVP_DecodeBlock() with caller-controlled lengths, bound the input below 1,610,612,733 bytes now, in your own code. Do not wait for the advisory.
  • Size base64 output buffers in size_t, never int. The documented formula is correct arithmetic in the wrong type.
  • Prefer the streaming EVP_EncodeUpdate() path for large inputs — it already refuses output over INT_MAX.
  • If you vendor OpenSSL, watch for the #33652 backports. At the time of writing, a backport to 3.6.5 was already circulating; 3.0.x, which is what Ubuntu 24.04 ships, had no fixed release yet.

The honest summary: this is a 30-year-old function with a 32-bit length in its signature, and base64's 4/3 expansion means the length stops fitting a few hundred megabytes before the input stops fitting one. The overflow was always one large input away. Somebody finally brought one.

Limitations, stated plainly

  • The ASan test compiled the scalar encoder verbatim from OpenSSL 3.0's source; it is not a rebuilt libcrypto. The system-library test used the real libcrypto.so.3 from Ubuntu 24.04 (OpenSSL 3.0.13).
  • I did not test the AVX2 encoder path — 3.0.13 predates it — and I did not build an exploit. The heap overflow is demonstrated; weaponization is not.
  • The fix PR was open, not merged, when I checked on the morning of November 20, 2026. If it merges with changes, the exact fix text may differ.

Sources: OpenSSL PR #33652, "Buffer overflow in EVP_EncodeBlock() (CVE-2026-121481)"; OpenSSL 3.0 crypto/evp/encode.c; Hacker News discussion, 322 points. All experiments run locally November 20, 2026: gcc 13.3.0, OpenSSL 3.0.13, 4-vCPU VM.