1.5 GiB in, 2.1 GiB out: OpenSSL's base64 overflows an int
One byte past 1,610,612,733 and EVP_EncodeBlock() writes 2,147,483,648 bytes into a buffer your own size math said was −2,147,483,647 bytes long. I ran it against the OpenSSL 3.0.13 on my machine: it returned −2,147,483,648, and AddressSanitizer caught the verbatim encoder writing past a 1 MiB buffer.
The bug is CVE-2026-121481, scored 9.8, disclosed this week in OpenSSL pull request #33652. EVP_EncodeBlock() takes an int dlen and returns an int — the length of the encoded output. Base64 expands 3 bytes to 4, so the output is 4 * ceil(dlen / 3) bytes. That only fits in a signed 32-bit int while dlen <= (INT_MAX / 4) * 3, which is 1,610,612,733 bytes — about 1.5 GiB. One byte more and the length wraps. The function never checks. It just keeps writing.
The threshold, computed
INT_MAX = 2,147,483,647 threshold (INT_MAX/4)*3 = 1,610,612,733 (1.500 GiB) dlen=1,610,612,733 -> int formula = 2,147,483,645, real bytes = 2,147,483,645, overflow = no dlen=1,610,612,734 -> int formula = -2,147,483,647, real bytes = 2,147,483,649, overflow = YES The "int formula" column is what a caller gets from the documented sizing rule, 4 * ((dlen + 2) / 3) + 1, computed in 32-bit int. At the threshold it is fine. One byte past it, the same expression wraps to a negative number. The encoder, meanwhile, really writes 2,147,483,649 bytes. The gap between those two numbers is the overflow.
Experiment 1: AddressSanitizer on the verbatim encoder
I took evp_encodeblock_int() verbatim from OpenSSL 3.0's crypto/evp/encode.c — the scalar loop, unchanged — compiled it with -fsanitize=address, and called it with dlen = 1,610,612,734 and a 1 MiB output buffer:
dlen=1610612734, out buffer=1 MiB, calling encoder... ==11787==ERROR: AddressSanitizer: heap-buffer-overflow WRITE of size 1 #0 in main 0x70b2c18f6800 is located 0 bytes after 1048576-byte region [0x70b2c17f6800,0x70b2c18f6800) SUMMARY: AddressSanitizer: heap-buffer-overflow in main The loop is almost comically simple — for (i = dlen; i > 0; i -= 3), four bytes out per iteration, ret += 4:
for (i = dlen; i > 0; i -= 3) { if (i >= 3) { l = (((unsigned long)f[0]) << 16L) | (((unsigned long)f[1]) << 8L) | f[2]; *(t++) = conv_bin2ascii(l >> 18L, table); *(t++) = conv_bin2ascii(l >> 12L, table); *(t++) = conv_bin2ascii(l >> 6L, table); *(t++) = conv_bin2ascii(l, table); } else { /* ... padding path, same 4-byte write ... */ } ret += 4; f += 3; } *t = '\0'; return ret; Nothing in that loop knows or cares how big the destination is. ret wraps to a negative number around iteration 536 million, and the writes just continue. The only thing that stops it in my test is ASan's redzone. In a real program, the only thing that stops it is the end of the heap.
Experiment 2: the shipped library, 1.5 GiB for real
The ASan test used a copy of the function, so I also ran the real thing: the system libcrypto, OpenSSL 3.0.13, linked normally. Input was 1,610,612,734 bytes of A in a file on disk, memory-mapped — the encoder reads it sequentially, so the page cache reclaims behind it and resident memory stays small. Output went to a 2,147,483,649-byte sparse file, also memory-mapped. (My VM has 8 GiB of RAM with about 500 MB free; allocating 3.7 GiB of buffers was not an option, which is why both sides are files.)
OpenSSL 3.0.13 30 Jan 2024, dlen=1610612734, calling EVP_EncodeBlock... returned -2147483648 (expected 2147483648, wraps to -2147483648) out[0..3]=QUFB out[2147483644..2147483648]=QQ== out[2147483648]=0 Three observations, all from the real library:
- The return value is −2,147,483,648. The true output length is 2,147,483,648. The
intreturn type cannot express it, so the function reports a negative length for a successful encode. - The output itself is correct base64: it starts
QUFBand endsQQ==followed by the NUL terminator, exactly as 1,610,612,734Abytes should encode. The scalar encoder does not corrupt data at this size — it just cannot tell you how much it wrote. - The encode took 20 seconds wall clock, 1.6 seconds of CPU. The rest was the kernel writing 2.1 GiB of output back to disk. The bug is not a performance problem; it is a bookkeeping problem with a 2 GiB blast radius.
So which callers actually overflow? The ones who do what the documentation tells them to. The man page says the destination must hold 4 * ((n + 2) / 3) + 1 bytes, and n is an int. Compute that in int and you get a negative size for any input over the threshold; pass the negative to malloc and you get NULL (or, with a cast, something worse); hand the result to EVP_EncodeBlock and it writes 2.1 GiB into it. The newer AVX2 path has the same shape — EVP_EncodeBlock returns (int)encode_base64_avx2(...), truncating a size_t result to int.
Notably, the streaming API already knew about this. EVP_EncodeUpdate() has an explicit guard:
if (total > INT_MAX) { /* Too much output data! */ *outl = 0; return 0; } The one-shot EVP_EncodeBlock() was the hole. Same library, same file, one function guarded and the other not.
The fix, and what it actually is
PR #33652, by nmouha, is open at the time of writing — not yet merged, and the formal advisory is still pending. The proposed fix adds two asserts to EVP_EncodeBlock():
OPENSSL_assert(dlen >= 0); OPENSSL_assert(dlen <= (INT_MAX / 4) * 3); plus a guard in EVP_DecodeBlock() (0 for empty input, −1 for negative length) and a corrected length check in http_client.c's base64encode(), where the old test was len > INT_MAX — reachable, per the backport notes, through a long s_client -proxy_pass password.
Two things about the fix are worth stating precisely, because both are easy to get wrong:
First, OPENSSL_assert is not compiled out in release builds. I checked the macro in include/openssl/crypto.h.in — it unconditionally calls OPENSSL_die() on failure. The NDEBUG that OpenSSL's Configure adds for release builds does not affect it. So the proposed fix turns a silent heap overflow into a loud, immediate abort. That is the right trade for a library function, and it is not the "asserts disappear in production" story.
Second, an abort is still not an error return. The PR's own comments point at follow-up issue #33812, which would move the check into the internal functions and return success/failure instead of dying. A caller handing EVP_EncodeBlock() a 2 GiB buffer today gets memory corruption; with this fix it gets a dead process. Better, and still not good API design — but you take the patch that stops the bleeding.
What to do about it
- If you call
EVP_EncodeBlock()orEVP_DecodeBlock()with caller-controlled lengths, bound the input below 1,610,612,733 bytes now, in your own code. Do not wait for the advisory. - Size base64 output buffers in
size_t, neverint. The documented formula is correct arithmetic in the wrong type. - Prefer the streaming
EVP_EncodeUpdate()path for large inputs — it already refuses output overINT_MAX. - If you vendor OpenSSL, watch for the #33652 backports. At the time of writing, a backport to 3.6.5 was already circulating; 3.0.x, which is what Ubuntu 24.04 ships, had no fixed release yet.
The honest summary: this is a 30-year-old function with a 32-bit length in its signature, and base64's 4/3 expansion means the length stops fitting a few hundred megabytes before the input stops fitting one. The overflow was always one large input away. Somebody finally brought one.
Limitations, stated plainly
- The ASan test compiled the scalar encoder verbatim from OpenSSL 3.0's source; it is not a rebuilt libcrypto. The system-library test used the real
libcrypto.so.3from Ubuntu 24.04 (OpenSSL 3.0.13). - I did not test the AVX2 encoder path — 3.0.13 predates it — and I did not build an exploit. The heap overflow is demonstrated; weaponization is not.
- The fix PR was open, not merged, when I checked on the morning of November 20, 2026. If it merges with changes, the exact fix text may differ.
Sources: OpenSSL PR #33652, "Buffer overflow in EVP_EncodeBlock() (CVE-2026-121481)"; OpenSSL 3.0 crypto/evp/encode.c; Hacker News discussion, 322 points. All experiments run locally November 20, 2026: gcc 13.3.0, OpenSSL 3.0.13, 4-vCPU VM.