The question was simple: "How do I rotate the staging API key?" The answer was in my runbook index, word for word. My RAG pipeline returned a passage about rotating log files instead.

I opened the chunk that should have won. The answer was there, in the last third of it. The embedding model had never seen that part. all-MiniLM-L6-v2 silently truncates at 256 tokens, and my chunks were sized in characters, so more than a third of them were being embedded from their first half only. No warning, no error, no log line. Just a vector that described a different paragraph than the one I stored.

TL;DR

  • sentence-transformers/all-MiniLM-L6-v2 has max_seq_length = 256. Anything past token 256 (including [CLS] and [SEP]) is dropped before pooling, without a warning.
  • LangChain's RecursiveCharacterTextSplitter measures chunk_size in characters by default, not tokens. 1,500 characters of technical English is often 350+ tokens.
  • In my corpus, 1,174 of 3,120 chunks (37.6%) were over the limit. Their vectors only represented the opening of each chunk.
  • Fix: split with the embedding model's own tokenizer, keep chunks under max_seq_length minus special tokens, and assert it in your ingestion code.
  • Raising max_seq_length to 512 does not crash, but the model was trained on short sequences, so measure retrieval before trusting it.

Why does all-MiniLM-L6-v2 truncate at 256 tokens?

all-MiniLM-L6-v2 truncates at 256 tokens because sentence-transformers ships it with max_seq_length = 256 in its config, and the tokenizer is called with truncation=True. The model is a small BERT-style encoder that was fine-tuned on short sentence pairs, so 256 was chosen as a safe ceiling.

You can see it in two lines:

from sentence_transformers import SentenceTransformer model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") print(model.max_seq_length) # 256

The important word is silently. model.encode() takes a 2,000-token string and a 200-token string and hands you back two 384-dimensional vectors that look equally legitimate. There is no flag on the output that says "this one was cut".

Other popular models have different ceilings. all-mpnet-base-v2 ships with 384. Many BGE and E5 variants use 512. Check model.max_seq_length for whatever you run instead of trusting a blog post (including this one).

What actually happens to the tokens past the limit?

They are deleted before the model runs. The tokenizer produces input_ids, keeps the first 255 or so plus the closing [SEP], and throws the rest away. The transformer never computes attention over them, and mean pooling averages only the surviving token vectors.

So your stored vector is an honest embedding of the first ~250 tokens. It's a dishonest embedding of the chunk you saved next to it. At query time, the retriever compares the question against that partial vector, finds nothing about key rotation, and ranks the chunk low. Then your LLM receives a top-5 that doesn't contain the answer and does its best impression of a confident colleague.

This is why the failure is so annoying to debug. Every component behaves correctly:

  • The splitter produced the chunk size you asked for.
  • The embedding model embedded exactly what it was allowed to see.
  • The vector store returned the true nearest neighbours.
  • The LLM answered from the context it got.

The bug lives in the gap between two units: the splitter counts characters, the model counts tokens.

How do I check if my RAG chunks are being truncated?

Tokenize every chunk with the embedding model's own tokenizer and count how many exceed max_seq_length. It takes a minute and needs no GPU.

tok = model.tokenizer limit = model.max_seq_length lengths = [len(tok(c, add_special_tokens=True)["input_ids"]) for c in chunks] over = [n for n in lengths if n > limit] print(f"{len(over)} of {len(chunks)} chunks over {limit} tokens") print(f"max length: {max(lengths)}, median: {sorted(lengths)[len(lengths)//2]}") print(f"tokens never embedded: {sum(n - limit for n in over)}")

My output, on 3,120 chunks from about 400 internal runbooks and READMEs:

1174 of 3120 chunks over 256 tokens max length: 611, median: 231 tokens never embedded: 98,447

Look at the median. 231 tokens sounds safe. It made me assume the splitter was fine for weeks. The problem was the tail: anything with code blocks, file paths, YAML, or long hostnames tokenizes badly. kubectl rollout restart deployment/payments-api -n staging is a single line of text and a pile of subword tokens. Technical docs are exactly the content that blows past the budget.

My splitter config looked reasonable:

from langchain_text_splitters import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter(chunk_size=1500, chunk_overlap=150)

1,500 characters of plain English prose lands somewhere around 300-375 tokens with WordPiece. Code-heavy text goes higher. Neither fits in 256.

Why didn't my retrieval evals catch it?

Because most answers sit near the top of a chunk. Headings, first sentences, and summaries come first in docs, and those are inside the 256-token window. My eval set of 120 questions scored well enough that I stopped worrying.

When I went back and located each answer span inside its gold chunk, 31 of the 120 answers started after token 256. Those were the questions that "the model was bad at". They weren't hard questions. They were questions whose answers had been cut out of the index.

If you build a retrieval eval, record where in the chunk the answer lives. A hit rate split by answer position (first 256 tokens vs. after) shows this bug in one table.

How do I fix chunk size for sentence-transformers models?

Size chunks in the embedding model's tokens, leave room for special tokens, and enforce the limit at ingestion time. LangChain can do the token counting for you if you pass it the right tokenizer:

from langchain_text_splitters import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter.from_huggingface_tokenizer( model.tokenizer, chunk_size=240, # 256 minus [CLS], [SEP], and some slack  chunk_overlap=32, ) chunks = splitter.split_text(doc)

Two details matter here:

  1. Use the embedding model's tokenizer, not tiktoken. A cl100k_base count and a WordPiece count for the same string can differ a lot on code. Counting with your LLM's tokenizer and embedding with BERT's is the same unit mismatch in a nicer outfit.
  2. The recursive splitter can still overshoot. If a single unbreakable piece (a giant minified line, a long table row) is bigger than chunk_size, it can come out oversized. So assert: 
for c in chunks: n = len(model.tokenizer(c)["input_ids"]) assert n <= model.max_seq_length, f"chunk is {n} tokens: {c[:80]!r}"

I'd rather have ingestion crash loudly than have retrieval degrade quietly.

After re-chunking at 240 tokens, my index went from 3,120 chunks to 4,388, and zero were over the limit. On the same 120 questions, the gold chunk appeared in the top 5 for 101 of them, up from 79. That's one corpus and one eval set on my machine, not a general benchmark, but the direction was not subtle.

Can I just set max_seq_length to 512?

You can, and it won't throw. MiniLM is a BERT-style model with 512 position embeddings, so model.max_seq_length = 512 runs fine. Whether it retrieves well is a separate question.

The model was fine-tuned on short text pairs. Asking it to pool 500 tokens into one vector takes it off the distribution it learned. Mean pooling over a long chunk also blurs topics together: a chunk covering key rotation, log retention, and on-call escalation becomes a vector that's vaguely about all three and strongly about none. I tried 512 on my eval set and it was worse than proper 240-token chunks, which matched my intuition but is still just my data.

If you need long chunks, switch to a model trained for long inputs and check its max_seq_length the same way. Don't stretch a sentence model and hope.

A checklist I now run on every RAG pipeline

  • Print model.max_seq_length and write it in the config next to chunk_size.
  • Confirm what unit your splitter uses. Default length_function=len means characters.
  • Count tokens per chunk with the embedding tokenizer. Look at the tail, not the median.
  • Assert the limit during ingestion.
  • Tag eval questions with the answer's token offset inside the gold chunk.
  • Re-check all of the above whenever you swap embedding models. A new model with a 512 limit makes your old chunks fine. Going the other direction breaks them silently.

So, does all-MiniLM-L6-v2 really truncate at 256 tokens?

Yes. all-MiniLM-L6-v2 silently truncates every input at 256 tokens (special tokens included) and embeds only that prefix, with no warning. If your text splitter measures chunks in characters, as LangChain's RecursiveCharacterTextSplitter does by default, a large share of technical chunks will exceed that limit and get embedded from their first half only, which makes RAG retrieval miss answers that sit later in the chunk. Fix it by splitting with the model's own tokenizer at roughly 240 tokens, asserting the limit during ingestion, and measuring retrieval by answer position. Raising max_seq_length to 512 runs, but it's no substitute for right-sized chunks or a long-context embedding model.

Written by the developer behind Preterview, an interview prep platform.