Take three sentences about Berlin, the same three Jina AI used to introduce late chunking:

  1. "Berlin is the capital and largest city of Germany, both by area and by population."
  2. "Its more than 3.85 million inhabitants make it the European Union's most populous city, as measured by population within city limits."
  3. "The city is also one of the states of Germany, and is the third smallest state in the country in terms of area."

Split them into one chunk per sentence, embed each chunk on its own, and score each one against the query "Berlin". In Jina's run the cosine similarities came out at 0.849, 0.708 and 0.753. Sentences two and three are about Berlin. They just never say the word. "Its" and "The city" point back at a sentence the embedding model never saw.

Run the same three chunks through late chunking and Jina reports 0.850, 0.825 and 0.850. Same model, same chunks, same query. The only thing that changed is when the cut happens.

Why the middle chunk forgets Berlin

The usual RAG pipeline splits first and embeds second. Each chunk goes into the embedding model as a separate input, so the transformer's attention only covers the tokens inside that chunk. When the model reads "Its more than 3.85 million inhabitants", the word "Berlin" is nowhere in its input. The token "its" gets a vector for a generic possessive pointing at some large place. Mean pooling averages that into the chunk vector, and the result lands closer to "big European city" than to "Berlin".

The late-chunking paper by Günther, Mohr, Williams, Wang and Xiao (first posted September 2024, revised July 2025) states the problem plainly: chunk embeddings built this way "can lose contextual information from surrounding chunks". Pronouns make a clean demo, but the same failure hits "the company", "this endpoint", "the second option", or any reference whose antecedent landed in a different chunk.

A paper document cut into three strips by scissors, with an orange thread from the middle strip snapping before it reaches a small city skyline
Naive chunking: each strip is embedded alone, so the middle one loses its link to the city.

Run the transformer once, cut afterwards

Most embedding models already work in two stages. The transformer turns every token into a vector that has attended to every other token in the input. Then a pooling step (mean pooling, for the Jina v2 family) collapses those token vectors into one vector for the whole input.

Late chunking moves the chunker between those two stages. Algorithm 1 in the paper's current version goes like this:

  1. Decide chunk boundaries the way you already do: fixed size, sentences, or a semantic splitter. You get character spans.
  2. Tokenize the whole document and run the transformer over all of it, or as much as fits in the context window.
  3. Convert each character span into a token span.
  4. Mean-pool the token vectors inside each token span. One vector per chunk.

Step 2 is where "Berlin" gets in. The token vectors for "its" were computed with attention over the first sentence, so they already carry the reference. Pooling them per chunk keeps that context inside each chunk vector. Jina's write-up puts it this way: late chunking "still requires boundary cues", but those cues "are used only after obtaining the token-level embeddings."

Two properties make it cheap to adopt. The abstract says the method "works without additional training" (the paper also offers an optional fine-tuning recipe on top). And the output doesn't change shape: one vector per chunk at the same dimension, so your vector store stays as it is.

A long scroll passing whole through a lens, then a blade slicing the resulting row of beads into three groups, each group tied by an orange thread back to a city skyline
Late chunking: the model reads the whole document first, and the cut happens on the token vectors.

Rebuilding it in Node with Transformers.js

Transformers.js runs ONNX models in Node and the browser. Xenova/jina-embeddings-v2-small-en is the ONNX port of jina-embeddings-v2-small-en, a 33M-parameter, Apache-2.0 model with an 8,192-token window. Its tokenizer config declares a BertTokenizer with model_max_length set to 8192.

npm i @huggingface/transformers

Setup and the document:

// late-chunk.mjs import { AutoTokenizer, pipeline, cos_sim, } from '@huggingface/transformers'; const MODEL = 'Xenova/jina-embeddings-v2-small-en'; const MAX_TOKENS = 8192; const chunks = [ 'Berlin is the capital and largest city of Germany, ' + 'both by area and by population.', 'Its more than 3.85 million inhabitants make it the ' + "European Union's most populous city, as measured " + 'by population within city limits.', 'The city is also one of the states of Germany, and ' + 'is the third smallest state in the country in ' + 'terms of area.', ]; const doc = chunks.join(' '); const tokenizer = await AutoTokenizer.from_pretrained(MODEL); const extractor = await pipeline( 'feature-extraction', MODEL, { dtype: 'fp32' }, );

Next, the token spans. The Transformers.js tokenizer options don't include an offset mapping like the Python tokenizers have, so this version counts tokens per chunk instead. The chunks are whole sentences joined by spaces, and a BERT tokenizer splits on whitespace before it breaks words into pieces, so tokenizing each chunk alone yields the same tokens as tokenizing the full document. The code still checks that the total token count matches what the model saw.

// Token span per chunk. Position 0 is [CLS]. function tokenSpans(chunks) { const spans = []; let cursor = 1; for (const text of chunks) { const ids = tokenizer.encode(text, { add_special_tokens: false, }); spans.push([cursor, cursor + ids.length]); cursor += ids.length; } return { spans, total: cursor + 1 }; // + [SEP] }

Now the late part. With pooling: 'none', the feature-extraction pipeline returns a tensor shaped [1, tokens, hidden], one vector per token. Pooling is a loop over a flat Float32Array:

function meanPool(data, dim, start, end) { const v = new Float32Array(dim); for (let t = start; t < end; t++) { const row = t * dim; for (let d = 0; d < dim; d++) v[d] += data[row + d]; } for (let d = 0; d < dim; d++) v[d] /= end - start; return Array.from(v); } async function lateChunk(doc, chunks) { const { spans, total } = tokenSpans(chunks); if (total > MAX_TOKENS) { throw new Error(`${total} tokens, window is ${MAX_TOKENS}`); } const out = await extractor(doc, { pooling: 'none' }); const [, seqLen, dim] = out.dims; if (seqLen !== total) { throw new Error(`spans cover ${total}, model saw ${seqLen}`); } return spans.map(([s, e]) => meanPool(out.data, dim, s, e)); }

And the comparison, with the naive version embedding each chunk as its own input:

async function naive(chunks) { const out = await extractor(chunks, { pooling: 'mean' }); return out.tolist(); } const q = await extractor('Berlin', { pooling: 'mean' }); const query = q.tolist()[0]; const before = await naive(chunks); const after = await lateChunk(doc, chunks); chunks.forEach((text, i) => { const a = cos_sim(query, before[i]).toFixed(3); const b = cos_sim(query, after[i]).toFixed(3); console.log(`naive ${a} late ${b} ${text.slice(0, 30)}`); });

Run it with node late-chunk.mjs. I haven't run this exact script, so there's no output table here to copy. What to check is the direction: chunks two and three should climb toward chunk one. Jina's published numbers came from their own Python code, and an ONNX port won't match them to the decimal. The model page's own example shows fp32 and quantized weights already disagreeing in the third decimal (0.940 against 0.934). One deliberate difference from the paper: its Algorithm 1 folds [CLS] and [SEP] into the first and last chunk. To match that, start the first span at 0 and end the last span at total.

Does it hold beyond a three-sentence toy? Jina ran BEIR retrieval tasks with roughly 256-token chunks on the small model and published nDCG@10:

Dataset Avg doc length (chars) Naive Late No chunking
SciFact 1,498 64.20% 66.10% 63.89%
TRECCOVID 1,117 63.36% 64.70% 65.18%
FiQA2018 767 33.25% 33.84% 33.43%
NFCorpus 1,590 23.46% 29.98% 30.40%
Quora 62 87.19% 87.19% 87.19%

Quora documents average 62 characters, so each one fits in a single chunk and late chunking has nothing to add. In this table, the longer the documents, the bigger the gap. On TRECCOVID and NFCorpus, not chunking at all scored slightly higher than late chunking. Averaged over three models and four BEIR datasets, the paper reports nDCG@10 going from 52.2 to 54.0 with fixed-size boundaries and from 52.4 to 54.3 with sentence boundaries.

The 8,192-token wall

The trick only works if the document fits in one forward pass. Jina describes the v2 window of 8,192 tokens as "roughly ten standard pages of text". The model card says the model was trained at 512 tokens and uses a symmetric bidirectional variant of ALiBi to extrapolate to about 8k. That's why the code throws past MAX_TOKENS instead of letting anything truncate quietly.

For longer documents, the paper's Algorithm 2, "long late chunking", splits the token sequence into macro chunks of the maximum length, each overlapping the previous one by a set number of tokens. The overlap tokens provide context to the window but aren't reused in the output. The token vectors from all windows are concatenated, then pooled per chunk as before. The limit is still there: a chunk in window three can't see a definition that only appears in window one.

A few other costs come with it:

Late chunking vs Anthropic's contextual retrieval

Anthropic published Contextual Retrieval on September 19, 2024, about two weeks after the paper's first version. Their method sends Claude 3 Haiku the full document plus one chunk and asks for "a short succinct context to situate this chunk within the overall document". The answer, usually 50 to 100 tokens, gets prepended to the chunk before embedding and before BM25 indexing. Anthropic measured retrieval failures (1 minus recall@20) dropping 35% with contextual embeddings, 49% when contextual BM25 is added, and 67% with reranking on top. They estimated the one-time cost at $1.02 per million document tokens, assuming 800-token chunks, 8k-token documents and prompt caching.

That is one LLM call per chunk. A corpus of 50,000 chunks means 50,000 generation calls at index time, plus more whenever documents change. Late chunking needs one forward pass of a 33M-parameter model per document, with no generation step. Jina's follow-up calls the LLM route "expensive in terms of cost, time, and storage", which is fair to weigh while remembering who wrote it.

The paper ran both on one small example (Section 4.5): a fictional financial document and a question about ACME Corp's Q2 2023 revenue growth. For the chunk "It highlighted a 3% revenue growth over the previous quarter.", naive chunking scored 0.6343, late chunking 0.8516, and contextual embeddings 0.8590. That's a single example: it shows the two methods landing close on one chunk and says nothing about a full corpus.

Contextual retrieval has real advantages. The added context is text, so it helps BM25 too, and it travels with the chunk into the prompt you send the generator. You can also tune the prompt for your domain, for example with a glossary, as Anthropic suggests. Late chunking only changes vectors: your keyword index and your LLM still see the raw "Its more than 3.85 million inhabitants".

A reasonable order: if you host your own embedding model and your documents fit in 8k tokens, try late chunking first, because it costs one code change at index time. Reach for contextual retrieval when you need keyword search to benefit from the context, or when an index-time LLM bill is acceptable for your corpus size. Nothing stops you from doing both, though I haven't seen published numbers for the combination.

If this was useful

Late chunking is a small change at index time that plugs a recall leak most pipelines don't measure. AI That Reads is the book in my series about that whole layer in TypeScript: chunking, embeddings, vector search, and grounding answers in your own documents. If your retrieval looks fine in demos and misses on real questions, it's written for you.

AI That Reads — AI in TypeScript

It's book 2 of AI in TypeScript, five books from your first LLM call to agents in production: the whole series is here.