Turning messy docs into structured Markdown using Azure Document Intelligence + Pydantic + Jinja2

How many times have you received a document that's messy, unstructured, or just plain hard to work with? Now imagine you need a whole stack of them as source material for an AI app. That was my task: Word documents, some laid out as decision trees, some as plain narratives. Structure tells a model what matters and how the pieces connect, so the goal was to turn those documents into a clean, structured Markdown format. To get there, I built a pipeline where Azure Document Intelligence extracts the content, Pydantic enforces the structure, and Jinja2 renders the Markdown. I also used Claude from Anthropic for every step that needed an LLM, but you can swap in any LLM you prefer.

The pipeline

flowchart TD A[.docx file] --> B["1. Extract text, tables and layout from the document<br/>using Azure Document Intelligence."] B --> C["2. Classify the document shape<br/>(decision tree or plain narrative)"] C --> E["3. Map the extracted text onto a Pydantic model,<br/>prompted with the few-shot examples<br/>for that shape."] E --> F["4. Validate the Pydantic model<br/>deterministically to catch structural issues."] F -->|validation flags something| G["5. Give the LLM one chance<br/>to repair it."] F -->|no issues| H["6. Render the Pydantic model<br/>to Markdown using Jinja2."] G --> R{"Re-validate: fewer warnings<br/>and no content lost?"} R -->|yes: keep the repaired version| H R -->|no: keep the original<br/>and its warnings| H H --> I["Structured Markdown (final.md)<br/>+ warnings.txt for human review"]

The pipeline runs in six stages. The Azure Document Intelligence prebuilt-layout model extracts text, tables, and layout from the source document. A classifier then determines whether the document is a decision tree or a plain narrative, and that label decides which few-shot examples go into the mapping prompt. The LLM maps the extracted text onto a Pydantic model, which is then validated deterministically to catch structural issues. If it flags a problem, the LLM gets one chance to repair it. If the repair doesn't help, the original version is kept for a human to review; otherwise, the repaired version moves on. Finally, the Pydantic model is rendered to Markdown using Jinja2.

The idea behind the design is that no stage trusts the one before it. Mapping rebuilds the structure instead of assuming extraction kept it, validation checks the mapping without asking an LLM, and a repair is only kept if the same checks, run again, show fewer warnings and no lost content.

For this pipeline, a document can be one of two shapes:

• a decision tree or
• a plain narrative

A decision tree is a branching structure, where each step leads to one or more outcomes.

"who is calling? → is the identity confirmed? → go to step 4"

A plain narrative is linear text, read top to bottom, with no branching.

"This document has information for..."

The goal is simple: turn either shape into a structured Markdown file that can be easily read and processed.

Before we start: the sample documents for the demo

The pipeline needs documents to run on, so the first thing to build was a small set of samples. They are written as generic "customer support call scripts", but the domain is arbitrary. The same pattern applies just as well to onboarding SOPs, IT runbooks, compliance procedures — anywhere structured and unstructured documentation coexist.

For this demo, I created 3 synthetic tree documents and 3 synthetic narrative documents, designed to represent the two document shapes the pipeline is built to handle:

• tree documents follow a decision-tree structure
• narrative documents are plain, linear text, read top to bottom.

Those six are then split two ways, and the split matters. Four of them stay in synthetic_docs/ and become the few-shot examples the model is shown at mapping time. They don't reach the model as Word files. The script src/create_few_shot_examples/create_raw_examples.py first runs each one through the same Azure extraction the pipeline uses and saves the result to the few_shot_examples/ folder. There, I paired each output with its gold standard. The remaining two, one of each shape, are held out in sample_inputs/ as the documents the demo actually processes. Nothing the pipeline runs on has ever been shown to the model as an answer.

Feel free to swap in your own documents immediately — this step is just there so you can run the demo out of the box, without needing to supply anything yourself.

Step 1: Extracting data with Azure Document Intelligence

Azure Document Intelligence reads a document and returns its text along with the layout it detects: headings, paragraphs, tables, and the boxes of a drawing. To run the demo, you'll need an Azure Document Intelligence resource, plus its endpoint and key.

The code snippets in this article are trimmed to the essentials: imports and logging are left out, and docstrings are shortened. The full code is in this repository.

endpoint = os.environ["AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT"] key = os.environ["AZURE_DOCUMENT_INTELLIGENCE_KEY"] 

Note: I built this whole demo on the free tier (F0), which gives you 500 pages a month at no cost. The F0 only reads the first two pages of each document, so if your file is longer, the rest is simply dropped. The sample documents are short, so it did not matter here, but longer files would require a paid tier. For details, see the Azure Document Intelligence pricing page.

In this demo, I am going to use two documents, one in a decision tree shape and one in a plain narrative shape. You can find these documents in the sample_inputs/ folder.

In order to extract the text from these documents, I will use the prebuilt-layout model of Azure Document Intelligence. It handles a wide range of layouts, and it can return its output as Markdown. That matters, because the layout it detects is encoded right in the text: section titles come back as # or ## headings instead of plain lines, and tables come back as pipe tables instead of a loose run of cell values. The next steps can then work with that structure instead of rebuilding it from scratch. If you want to go deeper on what the model can do, the Azure Document Intelligence documentation is a good next stop.

def extract_text(source_path: str | Path) -> str: """ Call Azure AI Document Intelligence (prebuilt-layout) on a real file. """ client = DocumentIntelligenceClient( endpoint=endpoint, credential=AzureKeyCredential(key) ) with open(source_path, "rb") as source_file: poller = client.begin_analyze_document( "prebuilt-layout", body=source_file, # output the extracted text in Markdown format  output_content_format=DocumentContentFormat.MARKDOWN, ) result = poller.result() return result.content 

Step 2: Classifying the document shape

The next step is the classification of the extracted text: is the document a decision tree or a plain narrative? The label determines which set of few-shot examples gets loaded into the LLM's prompt for the mapping step, and showing the model examples of the right shape is what helps it understand the document's structure and content.

Note: A Vision Language Model (VLM) could do this classification instead, since it sees what text extraction loses, like arrow direction and spatial layout. But a VLM costs more and needs an extra step to render the document as an image, so this demo classifies from the extracted text alone. If you'd rather use a VLM, feel free to swap in your own classifier here.

def classify(raw_text: str) -> str: """ Classify a document as "tree" or "narrative". """ client = get_client() # Anthropic client  response = client.messages.create( model=MODEL_NAME, max_tokens=16, # Keep it short, we only need a label.  thinking={"type": "disabled"}, system=CLASSIFY_PROMPT, # Tells the model how to classify.  messages=[{"role": "user", "content": raw_text}], ) # _first_text extracts the first text block from the response, which should contain the label.  label = _first_text(response).strip().strip(".").lower() print(f"Label is {label!r}") if label not in VALID_LABELS: raise ValueError(f"Unexpected classification label {label!r}.") return label 

Few-shot examples live in the few_shot_examples/ folder, organized one subfolder per category. Within each category, examples are paired by filename:

• a <name>_raw.txt file holding the example document,
• matched with a <name>_expected.json file holding what that document should map to.

Drop a new <name>_raw.txt / <name>_expected.json pair into a category folder, and the pipeline picks it up automatically. Just make sure every raw file has its matching expected file.

You might be wondering whether classifying first is really necessary, rather than just showing the model every few-shot example regardless of shape. Honestly, with only one or two examples per category, it isn't, because cost and latency won't be an issue either way. It's really up to you and what your project needs. That said, I'd still lean toward including it, since it means you won't have to rewrite the pipeline later if things scale up.

Step 3: Mapping onto a Pydantic model

Defining the data model

Pydantic is a data validation library for Python: you define your data as typed models, and it makes sure whatever comes in actually matches that shape. That's exactly what we need here, a way to catch inconsistencies or errors in the LLM's output early, before they ripple further down the pipeline.

Rather than two separate schemas, one for trees and one for narratives, I used a single CallScript model: the fields both shapes share, plus steps and outcomes for trees and main_text for narratives. A single validator then enforces the rule that ties it together: exactly one of steps or main_text is filled in, never both and never neither.

Why use Pydantic at all, when the LLM only needs a JSON input_schema? Function calling features in most LLMs already let you define a schema and get structured output, so why not just write that schema by hand? Because here the schema comes from Pydantic: the input_schema handed to the LLM is generated with CallScript.model_json_schema(). One class is both the schema the LLM sees and the validator that checks its answer, so the two can never drift apart. On top of that, Pydantic can enforce rules a JSON schema can't express, like the tree-or-narrative rule, which runs as a Python validator once the response arrives.

Every script has a few shared fields, plus a few that depend on its shape:

class CallScript(BaseModel): """ A CallScript describes either a decision-tree call flow or a plain-narrative procedure, never both. """ # common fields  title: str = Field(..., description="The name of the procedure.") category: str | None = Field( default=None, description="High-level grouping, e.g. 'Billing', 'Technical Support'.", ) attention_points: list[AttentionPoint] = Field( default_factory=list, description=( "Warnings or caveats the agent must not miss, preserved in the " "order and nesting they appeared in the source document." ), ) # Decision-tree shape  steps: list[DecisionStep] = Field(default_factory=list) outcomes: list[Outcome] = Field(default_factory=list) # Plain-narrative shape  main_text: str | None = Field( default=None, description="Free-form procedure text, used for non-tree documents.", ) @model_validator(mode="after") def check_single_shape(self) -> "CallScript": """ Enforce that a document is a tree OR a narrative, never both and never neither. """ is_tree = bool(self.steps) is_narrative = bool(self.main_text) if is_tree and is_narrative: raise ValueError( "A CallScript cannot have both `steps` and `main_text` set — " "pick one shape." ) if not is_tree and not is_narrative: raise ValueError( "A CallScript must have either `steps` or `main_text` set." ) return self 

Mapping the extracted text onto the model

With the text extracted and the shape classified, the next step is mapping that text onto the CallScript model. The LLM has no built-in knowledge of the model or its fields, so the request has to spell everything out. It includes:

• Instructions (the system prompt) describing the model's structure and how to map the text onto it
• Few-shot examples for the classified shape, passed as earlier conversation turns, so the model can see how similar documents were mapped before
• The extracted text of the document itself

The demo uses Claude, but any LLM that supports structured output will work. The output is then validated and turned into a CallScript object.

All the prompts used in the pipeline live in src/prompts/ folder. Keep in mind that they're built for this demo, not as a general-purpose template. The mapping prompt starts with the basics: never invent information and never drop it. Then it covers the trickier edge cases:

• Pick exactly one shape, tree or narrative. Never both, never neither.
• Make every branch point to a step or outcome that actually exists.
• Treat outcomes as endpoints. They never route anywhere else.
• Keep every sentence of narrative content, without summarizing.

def map_to_schema(raw_text: str, category: str) -> CallScript: """ Map raw text into a validated CallScript. """ few_shot = load_few_shot_examples(category) messages = [] for example_raw, example_expected in few_shot: # Each example becomes a user turn (its extracted text) and an  # assistant turn (its hand-written CallScript), as a worked answer.  messages.append({"role": "user", "content": example_raw}) messages.append({"role": "assistant", "content": example_expected}) # The document to map goes last: the user turn the model must answer.  messages.append({"role": "user", "content": raw_text}) # Forced tool use: Claude must return a CallScript, which is then validated.  # An incomplete reply raises an error.  return request_call_script( system=AGENT_PROMPT, messages=messages, tool_description="Record the CallScript structure extracted from the source text.", ) 

Note: Instead of forced tool use, you could use Claude's native structured outputs, which guarantee the response matches your schema.

Step 4: Validating the Pydantic model

LLMs can hallucinate or misread the source text, so trusting the mapping step's output blindly is risky. Instead, I added a deterministic validation step that checks the CallScript object before it reaches the rendered Markdown.

Each check is its own function, and which ones apply depends on the document's shape:

Note: Three of these checks compare wording, and all three use the same rule: two pieces of text match when at least 80% of the distinct words in one are found in the other. What differs is the direction:

• Narrative coverage and Attention-point coverage start from the source. For each source sentence or attention-point line, they ask: did it make it into the output?
• Ungrounded nodes starts from the output. For each step and outcome, it asks: does it come from a line in the source?

80% is loose enough to let the model reflow a paragraph or fix a typo, and strict enough to catch a summarized or dropped sentence.

The last check, Shape conflicts, is a safety net: the Pydantic model already rules them out, but a CallScript built with model_construct() skips validation, so it can still reach this step if someone builds one by hand.

def sanity_check(script: CallScript, raw_text: str) -> list[str]: """ Run all structural checks and return a list of warning strings. """ warnings: list[str] = [] # Narrative-only checks  if script.main_text: warnings.extend(_check_narrative_coverage(script, raw_text)) warnings.extend(_check_narrative_size(script, raw_text)) # Tree checks (no-ops on narratives, which have no steps or outcomes)  warnings.extend(_check_dangling_references(script)) warnings.extend(_check_unreachable_nodes_and_outcomes(script)) warnings.extend(_check_grounding(script, raw_text)) # Shape-agnostic checks  warnings.extend(_check_attention_coverage(script, raw_text)) # Shape-agnostic: this is a safety net for the case where a  # CallScript bypasses the Pydantic validator  warnings.extend(_check_shape_conflict(script)) return warnings 

The first two tree checks only look at the graph, so they can't tell a real box from one the model made up. If a branch points at an invented outcome, the reference resolves and the outcome is reachable, so the graph looks perfect. The _check_grounding goes back to the source: it compares every step and outcome against the extracted text, line by line. An invented box won't match any line, so it gets flagged.

Narratives have the same blind spot. _check_narrative_coverage walks the source and asks whether each sentence made it into main_text, so a sentence that exists only in the output is never examined, and its extra words can even help other sentences reach the threshold. The _check_narrative_size closes that gap with a simple count: main_text restates the source body word for word, so its size should never be larger than the body's.

Attention points have a blind spot too. Narrative coverage stops at the "Attention points" heading, and grounding only looks at steps and outcomes, so a dropped caveat would pass every check, even though caveats are exactly what the agent must not miss. The attention-point coverage check applies the same 80% threshold to every line under that heading, matching each one against a single attention point, nested ones included.

Finally, none of these checks stop the pipeline. Instead of raising an error, each one adds its warnings to a single list, and that list decides what happens next: if it's empty, the script goes straight to rendering; if not, the warnings go to the repair step, so the model knows what to fix.

Note: A script can pass every check and still be wrong. The validation step is not a guarantee of correctness, only a way to catch common structural issues that are easy to check for deterministically. It is still possible for the model to misinterpret the source text or make other mistakes that the checks don't catch. The goal is to catch the low-hanging fruit, so human review can focus on what the checks can't see.

Step 5: Repairing the Pydantic model

If validation flags anything, the LLM gets exactly one attempt to repair it: it's shown the warnings, along with the current object and the source text, and asked to fix them. I capped it at one try on purpose. Without a limit, the model could keep "fixing" the same problem in a loop, burning time and tokens without ever getting it right.

Here's the part worth calling out, because it's easy to get wrong: checking that the warning count went down isn't enough. Say the model gets a "dangling reference" warning. The easiest way to make it disappear is to simply delete the branch that causes it. The warning is gone, but the script has lost content, so it's actually worse than before.

That's why a repair is only accepted if it passes two checks:

1. It reduces the number of warnings.
2. It doesn't lose any content. The CallScript model's content_size property adds up every step, outcome, and branch in a tree, every word in a narrative, and every attention point in both, nested ones included. The pipeline compares it before and after the repair. If the number drops, something was deleted, and the repair is rejected.

@property def content_size(self) -> int: """ Rough size metric used to detect a repair that deletes content. """ branch_count = sum(len(step.branches) for step in self.steps) narrative_words = len(self.main_text.split()) if self.main_text else 0 attention_count = len(flatten_attention_points(self.attention_points)) return ( len(self.steps) + len(self.outcomes) + branch_count + narrative_words + attention_count ) 

Note: At the end of the repair step, the remaining warnings are stored in a warnings.txt file in the outputs/ folder alongside the final Markdown output. The file only covers what's left after repair: if a repair was accepted, the warnings it fixed aren't recorded anywhere. For a full history of what was flagged and fixed, you'd need to add your own logging around attempt_repair's return values.

Putting it together, the repair step sends the source text, the mapped CallScript and its warnings back to the LLM, validates the reply into a new CallScript, and runs the checks on it again. If it passes both checks above, it replaces the original; otherwise, the original is kept. Either way, the pipeline moves on to rendering, and any warnings that remain end up in warnings.txt.

def attempt_repair( script: CallScript, warnings: list[str], raw_text: str ) -> tuple[CallScript, list[str], bool]: """ Attempt exactly one repair call and apply the acceptance gate. """ if not warnings: return script, warnings, False # Repair the script and re-validate it.  candidate = llm_repair(script, warnings, raw_text) candidate_warnings = sanity_check(candidate, raw_text) # check that the repair reduced the number of warnings and didn't lose content  warnings_improved = len(candidate_warnings) < len(warnings) size_preserved = candidate.content_size >= script.content_size if warnings_improved and size_preserved: return candidate, candidate_warnings, True return script, warnings, False 

Note: Why does the repair step need the original text? Some warnings can be fixed from the object alone: if a branch points to a step or outcome that doesn't exist, the model can see that and repair it on the spot. But if the model dropped a paragraph of narrative text, nothing in the object shows what went missing. Only the source does, which is why the repair prompt includes it.

You'll see a repair at work on a real document in the decision tree example below.

Step 6: Rendering the Pydantic model to Markdown with Jinja2

The last step turns the CallScript model into Markdown with Jinja2, a Python templating engine that lets you define exactly how each field of the model is written out — layout, formatting, structure, all of it. The same template handles both decision trees and plain narratives, so there's no need to maintain two versions. Rendered this way, the output ends up structured underneath and easy to read on top.

The template for this demo outputs the title, category and attention points unconditionally. Even when a section is empty, it still renders with a placeholder like "None." rather than being skipped. That's deliberate: an empty section is a recorded fact ("this document has no attention points"), while an omitted one would be indistinguishable from something the mapping step simply missed.

From there, the template has two shape-specific sections:

• when script.steps is set, it outputs the call flow and outcomes (the tree shape);
• when script.main_text is set, it outputs the narrative text.

The template checks the two conditions separately rather than as an if/else. The schema already guarantees exactly one is set, so this only matters if a CallScript skips validation, for example one built with model_construct(): the template still renders whatever it's given instead of breaking.

You can find the template in templates/call_script.md.j2.

def render_markdown(script: CallScript) -> str: """Render a CallScript into final Markdown using the Jinja2 template.""" jinja_env = Environment( loader=FileSystemLoader(TEMPLATES_DIR), trim_blocks=True, lstrip_blocks=True, ) template = jinja_env.get_template("call_script.md.j2") rendered = template.render(scriptSource：https://dev.to/agileactors/turning-messy-docs-into-structured-markdown-using-azure-document-intelligence-pydantic-jinja2-11mj
My favorites

Related posts

Three dates that catch a hard-coded time-zone bug in a session clock

A session that starts at 08:00 local time does not necessarily start at the same UTC hour all year. Less obviously, the

0dev.to

Laravel N+1 Query Problem: How to Find and Fix It

When a Laravel page that used to load in half a second suddenly takes four, the cause is very often the same: the N+1 qu

0dev.to

LongCat-2.5 at $0.30/$1.20: Thinking Locks Temperature, 1M Context, and Vendor Benches Between Opus 4.8 and 5

Attributed compile + one small offline run on my box Primary source: Introducing LongCat-2.5 — Native Multimodality for

0dev.to

Shipped: two packages, a registry listing, and the first seals

What happened Post 1 ended with a list: the verify package inside Fiador — human-backed attestation for Discord bots — i

0dev.to

AI Agent Memory Needs Garbage Collection, Not Just Storage

TL;DR — Most long-term memory systems for AI agents only know how to write and append — never to forget, consolidate, or

0dev.to

Securing the Analytics Pipeline and Pivoting for Demo Day

Sprint Retrospective: Implementing Zero-Trust Analytics The conclusion of Sprint 11 was entirely focused on access contr

0dev.to