Thursday AI Tool/Workflow. This week's tool: Docling, the open-source document converter started by IBM Research Zurich, now hosted by the LF AI & Data Foundation. It turns PDFs, DOCX, PPTX, HTML and more into clean Markdown or lossless JSON, and it can chunk the result for RAG without throwing away your section headings.

If your RAG pipeline starts with pdftotext or a basic PDF text extractor, you've probably seen what comes out: two-column pages read straight across, tables flattened into a stream of numbers, page headers and footers mixed into body text, and no idea which section a sentence came from. Then you split that text every N characters and wonder why retrieval is weak.

Docling attacks the problem at the source. Here's exactly how to use it.

What Docling gives you

From the official README, the features that matter for RAG:

  • Parsing of many formats, including PDF, DOCX, PPTX, XLSX, HTML, EPUB, images and more
  • Advanced PDF understanding: page layout, reading order, table structure, code, formulas and image classification
  • OCR support for scanned PDFs and images
  • A unified DoclingDocument representation, exportable to Markdown, HTML, DocTags and lossless JSON
  • Local execution, which matters for sensitive or air-gapped data
  • Integrations with LangChain, LlamaIndex, Crew AI and Haystack
  • MIT-licensed codebase (individual models have their own licenses)

1. Install 

pip install docling

Docling needs Python 3.10 or higher (Python 3.9 support was dropped in version 2.70.0). It runs on macOS, Linux and Windows, on both x86_64 and arm64.

2. Convert a document from the CLI

The fastest way to see what Docling does:

docling https://arxiv.org/pdf/2206.01062

This writes a .md file into the current directory. The CLI takes local files, directories or URLs, and a few options are worth knowing:

# Markdown + lossless JSON into ./out docling ./policies --to md --to json --output ./out # Scanned documents: OCR is on by default; turn it off for born-digital PDFs to save time docling report.pdf --no-ocr # Table structure model in accurate (default) or fast mode docling report.pdf --table-mode fast # Ask the CLI for chunks directly (hybrid chunker by default) docling report.pdf --to chunks --chunks-max-tokens 512

Run docling --help or check the CLI reference for the full list. The default chunk tokenizer for --to chunks is sentence-transformers/all-MiniLM-L6-v2.

3. Convert in Python

The Python API is three lines:

from docling.document_converter import DocumentConverter converter = DocumentConverter() result = converter.convert("policies/leave-policy.pdf") # local path or URL doc = result.document print(doc.export_to_markdown())

doc is a DoclingDocument. Besides Markdown you can call export_to_dict(), export_to_text(), export_to_doctags(), or save lossless JSON with doc.save_as_json(path). The JSON keeps structure and provenance that Markdown drops, so it is the better format to store if you might re-chunk later.

4. Tables and layout

Tables are where naive extraction hurts most. Docling runs a table structure model and gives you each table as a real object:

for i, table in enumerate(doc.tables): df = table.export_to_dataframe(doc=doc) # pandas DataFrame  df.to_csv(f"table-{i + 1}.csv")

In Markdown output, tables come out as Markdown tables instead of loose numbers. If you need to tune the PDF pipeline, pass pipeline options:

from docling.datamodel.base_models import InputFormat from docling.datamodel.pipeline_options import PdfPipelineOptions from docling.document_converter import DocumentConverter, PdfFormatOption opts = PdfPipelineOptions() opts.do_ocr = False # born-digital PDFs don't need OCR opts.do_table_structure = True # keep table structure  converter = DocumentConverter( format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=opts)} )

5. Chunk with HybridChunker

Docling has native chunkers that work on the DoclingDocument instead of on flat text. The docs describe HybridChunker as tokenization-aware refinements on top of document-based hierarchical chunking:

  • It starts from the hierarchical chunker (one chunk per document element, with headings and captions attached).
  • It splits chunks that are oversized for your tokenizer.
  • It merges undersized neighbouring chunks that share the same headings (merge_peers, on by default).
  • For tables that span chunks, repeat_table_header (on by default) repeats the header row in each chunk.

The docs stress one thing: the chunker and your embedding model should use the same tokenizer.

from docling.chunking import HybridChunker from docling_core.transforms.chunker.tokenizer.huggingface import HuggingFaceTokenizer from transformers import AutoTokenizer EMBED_MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2" tokenizer = HuggingFaceTokenizer( tokenizer=AutoTokenizer.from_pretrained(EMBED_MODEL_ID), max_tokens=256, # pick to fit your embedding model ) chunker = HybridChunker(tokenizer=tokenizer) for chunk in chunker.chunk(dl_doc=doc): print(chunk.meta.headings) # e.g. ["Leave Policy", "Carry-over"]  text_to_embed = chunker.contextualize(chunk=chunk)

Note contextualize(): it returns the chunk text enriched with metadata such as its headings. The docs say this is typically the text you want to feed the embedding model, while chunk.text is the raw content. If you prefer OpenAI tokenizers, install docling-core[chunking-openai] and use OpenAITokenizer with tiktoken.

6. Example workflow: a folder of policy PDFs into a RAG index

This is an example workflow, not a benchmark. Say HR has a folder of policy PDFs and you want a Q&A bot that cites the right section.

from pathlib import Path from docling.chunking import HybridChunker from docling.datamodel.base_models import ConversionStatus from docling.document_converter import DocumentConverter converter = DocumentConverter() chunker = HybridChunker() # default tokenizer; align it with your embedder in production  pdfs = sorted(Path("policies").glob("*.pdf")) Path("parsed").mkdir(exist_ok=True) records = [] for res in converter.convert_all(pdfs, raises_on_error=False): if res.status != ConversionStatus.SUCCESS: print("needs review:", res.input.file.name, res.status) continue doc = res.document doc.save_as_json(Path("parsed") / f"{res.input.file.stem}.json") # keep lossless copy  for i, chunk in enumerate(chunker.chunk(dl_doc=doc)): records.append({ "id": f"{res.input.file.stem}-{i}", "text": chunker.contextualize(chunk=chunk), # embed this  "source": res.input.file.name, "section": " > ".join(chunk.meta.headings or []), }) # embed records[*]["text"] and upsert into your vector store # (pgvector, Qdrant, Milvus...), storing source + section as metadata

What this buys you:

  1. Section headings travel with every chunk. "Carry-over" under "Leave Policy" doesn't get confused with "Carry-over" under "Expense Policy".
  2. Citations are free. Store source and section, and your answer can say "Leave Policy > Carry-over" instead of "page text, somewhere".
  3. Failures are visible. convert_all(..., raises_on_error=False) keeps going, and you log the files that only partially converted instead of silently indexing garbage.
  4. You can re-chunk without re-parsing. The saved JSON is a DoclingDocument, so changing chunk size later doesn't mean running the PDF models again.

Tips before you ship

  • Measure retrieval, not vibes. Build 20 to 50 real questions with known answer sections and check recall@k before and after switching to Docling.
  • Turn OCR off for born-digital PDFs and on for scans. It is a real speed/quality trade-off.
  • Spot-check tables. Open a few Markdown exports next to the original PDF, especially for merged cells.
  • Match tokenizers. Chunk limits only mean something if they're counted with your embedding model's tokenizer.

Garbage in, garbage retrieved. Fix the parsing step and every step after it gets easier.

References

What does your RAG pipeline use to parse PDFs today, and where does it break?