Pydantic for LLM Structured Output

TL;DR: This article explains how to use Pydantic to get strictly structured JSON output from Large Language Models. Readers will learn to define Pydantic models, prompt LLMs for structured data, and validate the results for robust AI applications.

Integrating large language models (LLMs) into software systems often hits a snag: LLMs are fantastic at generating human-like text, but when your application needs a specific data format, like JSON, they can be unreliable. You might ask for JSON, only to get a slightly malformed string, a missing field, or an unexpected data type, leading to frustrating parsing errors and fragile integrations.

This is where Pydantic comes in. It's a Python library for data validation and settings management that allows you to define data schemas using standard Python type hints. When you combine Pydantic with LLMs, you can enforce strict output formats, ensuring that the data you receive is always predictable and programmatically usable. Frankly, I think most AI agent projects break in production because of weak validation and error handling, not because of the underlying model. This technique directly addresses that.

Setting Up Your Environment

First, you'll need Pydantic. If you don't have it, a quick pip command handles it. Pydantic v2.14.3 is the latest stable release at the time of writing, offering performance improvements and new features over v1.

pip install pydantic

For interacting with LLMs, you'll typically use an SDK like OpenAI's or a framework like LangChain. While the direct LLM interaction part isn't strictly Pydantic, the principle is the same: you'll instruct the LLM to return JSON that matches your defined schema.

Defining Your Output Schema

The core of this technique involves defining a Pydantic model that represents the exact structure and types you expect from the LLM. Think of this as a contract for your data.

Let's say you're building a system that extracts key details from product reviews. You might want the LLM to return the product name, the sentiment of the review, and any detected features.

from pydantic import BaseModel, Field from typing import List, Literal, Optional class ReviewAnalysis(BaseModel): product_name: str = Field(..., description="The name of the product being reviewed.") sentiment: Literal["positive", "negative", "neutral"] = Field(..., description="The overall sentiment of the review.") features_mentioned: List[str] = Field(default_factory=list, description="A list of product features explicitly mentioned in the review.") reviewer_rating: Optional[int] = Field(None, ge=1, le=5, description="The numerical rating given by the reviewer, if any, on a scale of 1 to 5.") # You can even nest models for more complex structures class ProductOverview(BaseModel): product_id: str analyses: List[ReviewAnalysis]

In this example, ReviewAnalysis specifies that product_name must be a string, sentiment can only be "positive", "negative", or "neutral", features_mentioned is a list of strings (defaulting to an empty list), and reviewer_rating is an optional integer between 1 and 5. Pydantic handles type checking, default values, and even value constraints (like ge=1, le=5 for "greater than or equal to 1" and "less than or equal to 5").

Guiding the LLM and Parsing Output

Once you have your Pydantic model, the next step is to tell the LLM exactly what you want. Many modern LLM APIs and frameworks (like LangChain) have built-in support for Pydantic or JSON Schema. They can automatically generate a schema from your Pydantic model and pass it to the LLM.

If your LLM integration doesn't have direct Pydantic support, you can still get a JSON schema from your model using ReviewAnalysis.model_json_schema() and include it directly in your prompt. This tells the LLM the exact JSON structure it should return.

Here's an illustrative sketch of how you might prompt an LLM and then parse its response (not using a specific LLM SDK, but showing the conceptual flow):

import json # ... (ReviewAnalysis model definition from above) ... # This is the prompt you'd send to the LLM prompt = f""" Analyze the following product review and extract the specified information as a JSON object. Adhere strictly to the following JSON schema: {ReviewAnalysis.model_json_schema_json(indent=2)} Product Review: "This vacuum cleaner is amazing! The suction power is incredible (5/5 stars), and it's surprisingly quiet. The only downside is the battery life, which is a bit short. Definitely a positive experience overall." """ # Imagine 'llm_response_text' is the raw string output from your LLM # Example of a good response: llm_response_text_good = """ { "product_name": "Vacuum Cleaner", "sentiment": "positive", "features_mentioned": ["suction power", "quiet operation", "battery life"], "reviewer_rating": 5 } """ # Example of a potentially bad response (e.g., LLM hallucinated 'color') llm_response_text_bad = """ { "product_name": "Vacuum Cleaner", "sentiment": "positive", "features_mentioned": ["suction power"], "color": "blue", "reviewer_rating": 5 } """ try: # Attempt to parse the LLM's raw text response using Pydantic parsed_data = ReviewAnalysis.model_validate_json(llm_response_text_good) print("Successfully parsed data:") print(parsed_data.model_dump_json(indent=2)) print(f"Product name: {parsed_data.product_name}") print(f"Sentiment: {parsed_data.sentiment}") # You can now safely access attributes with correct types except Exception as e: print(f"Validation Error: {e}") try: # This will raise a ValidationError because 'color' is not in the schema parsed_data_bad = ReviewAnalysis.model_validate_json(llm_response_text_bad) except Exception as e: print(f"\nCaught expected validation error for bad response: {e}") # cite: 12 # Specific error handling for structured output issues is crucial, # as I discussed in [Validating AI Agent Outputs with Explicit Pass Conditions](https://dev.to/koolkamalkishor/validating-ai-agent-outputs-with-explicit-pass-conditions-5hm)

When using model_validate_json(), Pydantic attempts to parse the JSON string and then validates it against your ReviewAnalysis model. If the data types don't match, or if required fields are missing, or if extra unexpected fields are present, Pydantic raises a ValidationError with clear details. This means you get explicit failure signals instead of silent errors further down your processing pipeline.

A Practical Limitation

While Pydantic is powerful for ensuring structured output, it's not a magic bullet. The LLM still needs to be capable enough to follow the instructions and generate some form of valid JSON. Smaller or less capable models might struggle more consistently, leading to more frequent ValidationError exceptions. In such cases, you might need to iterate on your prompt engineering or consider using a more robust LLM. It's a trade-off between model capability and the strictness you need for downstream systems.

Sources