The claim that needs examining

A common framing in RAG tutorials goes like this: fixed-size splitting is a naive baseline, semantic chunking is the smarter upgrade, and agent-based (propositional) chunking is the frontier. Each step sounds like a clean improvement. The reality in production retrieval pipelines is more complicated, and the "upgrade" path often introduces failure modes that are harder to debug than the ones it replaces.

This post walks through the four main chunking families, where each one actually breaks, and what that means for choosing a strategy on a real corpus.

Four chunking strategies and their real tradeoffs

Rule-based splitting

This covers fixed-character splitting, fixed-token splitting (using the same tokenizer as the target model), content-aware splitting with tools like spaCy or NLTK, recursive splitting by structural hierarchy (sections, then paragraphs, then sentences), and format-specific splitting for Markdown, LaTeX, or HTML.

The standard criticism is that rule-based methods lack semantic understanding. That is true. But their failure modes are predictable and visible. A chunk that cuts mid-sentence because it hit a token limit is a problem you can see in your index. A chunk boundary placed by a miscalibrated similarity threshold is a problem you might only notice when recall drops and you cannot explain why.

For structured documents -- technical documentation, legal text, Markdown with clear heading hierarchies -- recursive rule-based splitting already respects topic boundaries. The heading structure is a chunking signal that an embedding model has to rediscover. Using it directly is not a limitation; it is using the right tool.

The key parameters are chunk_size and overlap. Overlap is not optional: without it, a query that spans two adjacent chunks will miss context that appears at a boundary. A 100-token overlap on 300-token chunks is a reasonable starting point for dense retrieval, though the right values depend on your retriever and corpus.

Semantic (embedding-based) splitting

The mechanics: embed each sentence, compute cosine similarity between adjacent sentences using a sliding window, and place chunk boundaries where similarity drops below a threshold. LlamaIndex's SemanticSplitterNodeParser is the most-cited implementation.

The appeal is real. On unstructured prose with no formatting signals -- long-form narrative, transcripts, some medical notes -- paragraph breaks are often arbitrary, and embedding-based boundary detection finds genuine topic shifts that rules would miss.

The problem is the threshold hyperparameter. There is no principled default. On a typical document corpus, moving the threshold by 0.05 can change the total chunk count by 30-40%. That means:

  • Your chunk size distribution is unstable across re-indexing runs
  • Swapping embedding models (e.g., upgrading from one version to another) silently changes chunk boundaries
  • Your retriever, which implicitly learned from a certain chunk length distribution, sees different inputs without any signal that anything changed

This does not mean semantic splitting is wrong. It means it requires more careful calibration and monitoring than it is usually presented as needing. Treat the threshold as a first-class hyperparameter, evaluate its effect on retrieval metrics (not just chunk aesthetics), and pin your embedding model version.

ML model-based splitting

Methods in this family use transformer models to predict segmentation points directly. The key variants:

Naive BERT uses next-sentence prediction (NSP) to score whether two adjacent sentences are consecutive, identifying boundaries where NSP confidence drops.

Cross Segment Attention combines BERT with a bidirectional LSTM to incorporate broader context around each candidate boundary, rather than evaluating sentence pairs in isolation.

SeqModel encodes multiple sentences simultaneously with BERT before predicting segmentation, capturing longer-range dependencies. It uses an adaptive sliding window during inference to keep latency manageable without sacrificing accuracy.

These methods can outperform rule-based and embedding-similarity approaches on unstructured text, particularly when the training data matches the target domain. The cost is deployment complexity: you are now running a separate model in your indexing pipeline, with its own versioning, inference cost, and domain adaptation requirements. For most production RAG systems, this overhead is justified only when the corpus is large enough and the retrieval quality gap is measurable.

Agent-based (propositional) splitting

The idea: rather than splitting by position, have an LLM decompose the document into atomic propositions -- independent factual statements, each representing a single claim. Then cluster propositions by semantic relatedness into chunks.

The steps are: (1) LLM reads and understands the text, (2) generates a list of propositions, (3) evaluates each proposition's relationship to existing clusters, (4) assigns it to an existing chunk or opens a new one, (5) iterates until the document is covered.

This produces very clean, semantically coherent chunks. For a small, high-value corpus -- a product knowledge base, a curated set of technical specifications -- the quality improvement is real.

For everything else, the cost is prohibitive. Each document requires multiple LLM calls at index time. On a corpus of thousands of documents, this is slow and expensive. The approach is genuinely interesting as a direction, but describing it as a production-ready strategy without those caveats is misleading.

The "long context replaces RAG" question

Some current models support context windows large enough to hold entire document collections. Kimi, for instance, supports a context of approximately 2 million Chinese characters. The argument: if you can fit everything in context, why retrieve at all?

Long context windows do provide real benefits: better coherence on complex multi-step tasks, the ability to reference background that a retriever might not rank highly, and a more forgiving environment when retrieval precision is imperfect.

But the argument for replacement fails on two points.

First, cost and latency. Running inference over 2M tokens is not cheaper than running a well-tuned retriever plus targeted generation over a few thousand tokens. For applications with high query volume, the economics do not work.

Second, and more importantly: a long context window does not fix retrieval; it tolerates bad retrieval. When the answer to a query depends on a specific value in a table, a clause in a contract, or a number in a technical spec buried in a 500-page document, the model still has to attend to the right passage. Long context helps when the relevant information is prominent or repeated. It does not help when it is sparse and specific -- exactly the case where a good retriever earns its place.

The more accurate framing: long context windows lower the bar for retrieval precision in some tasks, which simplifies certain RAG pipelines. They do not eliminate the need for retrieval on precision-critical tasks.

Choosing a strategy

The decision mostly comes down to corpus structure and scale:

  • Structured documents with clear formatting (Markdown, HTML, LaTeX, technical manuals): start with recursive rule-based splitting on structural boundaries. The formatting is already a chunking signal.
  • Unstructured prose at scale: semantic splitting is worth the calibration cost. Pin the threshold and the embedding model version, and monitor chunk size distribution across re-indexing runs.
  • Small, high-value corpus where quality dominates cost: agent-based splitting is worth evaluating.
  • Domain-specific structured prediction needs: ML model-based splitting, if you can afford the deployment overhead and have in-domain training data.

Overlap matters regardless of strategy. A chunk that is semantically clean but cuts off context at its boundary will still miss queries that span that boundary. Overlap is not a patch for bad chunking; it is a structural feature of any chunking scheme.

What specific failure mode made you move away from your first chunking approach, and what did you measure to confirm the switch was worth it?