Structured Outputs with LLMs: Validating AI Data with Pydantic

Reading an invoice, a contract or an email and returning the right fields is one of the most useful things a language model can do for a business. It is also where AI projects quietly fail: the model returns text instead of data, a field is missing, or everything looks right except the total.

This guide shows the three layers we put between a model and your database: a schema the model must follow, business rules checked in code, and a route to a person for anything that doesn't pass. The example is in Python with Pydantic and Claude; the same design works with OpenAI.

JSON mode is not enough

Asking a model to "reply in JSON" gets you JSON most of the time. OpenAI's documentation is explicit about the limit of its older JSON mode: it "will not guarantee the output matches any specific schema, only that it is valid and parses without errors."

Both major providers now offer something stronger:

• OpenAI Structured Outputs "ensures the model will always generate responses that adhere to your supplied JSON Schema." In the Python SDK you can pass a Pydantic model directly, with client.responses.parse(..., text_format=Model) or client.chat.completions.parse(response_format=Model).
• Claude structured outputs "guarantee schema-compliant responses through constrained decoding." You set output_config.format with a JSON schema, or in Python call client.messages.parse(..., output_format=Model) and read response.parsed_output. There is also strict tool use (strict: true on a tool) when the model should call a function with validated arguments.

Use these instead of prompting for JSON. They remove a whole class of errors: missing keys, invented enum values, broken syntax.

What a schema can't guarantee

A schema guarantees the shape of the answer. It does not guarantee that the answer is right.

• The response can stop early. Both providers document that output may not match the schema after a safety refusal or when the response hits the max_tokens limit. Claude reports this in stop_reason ("refusal" or "max_tokens").
• Some constraints aren't enforced by the API. Claude's structured outputs don't support numeric constraints such as minimum and maximum, or string minLength and maxLength; the Python SDK removes them from the schema it sends and checks them on the response instead. OpenAI supports a subset of JSON Schema too, and requires every field to be listed as required.
• Values can be well-formed and wrong. A total can be a valid number that doesn't match the line items. An invoice number can be a valid string the model made up.

So the schema is the first layer, not the whole solution.

Layer 1: a plain schema for the model

Keep the model-facing schema simple: field names, types, enums, nesting. Pydantic generates the JSON schema for you (model_json_schema()), and the provider SDKs accept the model class directly.

from datetime import date
from typing import Literal
from pydantic import BaseModel

class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: float
    amount: float

class InvoiceData(BaseModel):
    supplier_name: str
    invoice_number: str
    issue_date: date
    currency: Literal["USD", "EUR", "GBP"]
    line_items: list[LineItem]
    total: float

Literal becomes an enum the model can't step outside of. date becomes a date-formatted string that Pydantic turns into a real date.

Layer 2: business rules in code

The rules that make the data trustworthy live in a second model that inherits the first. Pydantic's Field constraints (gt, ge, min_length) and model_validator cover most of them, and any failure raises a ValidationError you can act on.

from pydantic import Field, model_validator

class CheckedLineItem(LineItem):
    quantity: float = Field(gt=0)
    unit_price: float = Field(ge=0)

    @model_validator(mode="after")
    def amount_matches(self):
        if abs(self.quantity * self.unit_price - self.amount) > 0.01:
            raise ValueError(f"{self.description}: quantity x unit price != amount")
        return self

class CheckedInvoice(InvoiceData):
    line_items: list[CheckedLineItem] = Field(min_length=1)
    total: float = Field(gt=0)

    @model_validator(mode="after")
    def total_matches(self):
        if abs(sum(i.amount for i in self.line_items) - self.total) > 0.01:
            raise ValueError("line items do not add up to the total")
        return self

Keeping the two apart matters. The model only has to fill in a shape; your code decides what counts as correct, and you can change those rules without touching the prompt.

Layer 3: grounding and human review

Models don't hand you a reliable confidence score. So "low confidence" has to be defined by checks you can run:

• The response stopped early: any stop_reason other than a normal end.
• A business rule failed: the ValidationError from layer 2.
• A value isn't in the source: some fields must appear verbatim in the document (an invoice number, an IBAN, a policy number). If it isn't there, the model may have invented it.

Anything that fails goes to a person instead of into your system. On OpenAI you can add another signal: the API can return token log probabilities (logprobs), which some teams use to flag uncertain fields.

The full example

This is the complete function. It returns either a checked invoice or a reason to send the document to review.

import anthropic
from pydantic import ValidationError

client = anthropic.Anthropic()

def extract_invoice(document_text: str):
    """Return (invoice, None) when the result can be trusted, or (None, reason) for human review."""
    try:
        response = client.messages.parse(
            model="claude-opus-5",
            max_tokens=16000,
            messages=[{
                "role": "user",
                "content": "Extract the invoice fields from this document. "
                           "Copy the invoice number exactly as written.\n\n" + document_text,
            }],
            output_format=InvoiceData,
        )
    except ValidationError as e:
        return None, f"output did not match the schema: {e.errors()[0]['msg']}"

    # A refusal or a cut-off response may not match the schema: never pass it on.
    if response.stop_reason != "end_turn" or response.parsed_output is None:
        return None, f"incomplete response ({response.stop_reason})"

    try:
        invoice = CheckedInvoice.model_validate(response.parsed_output.model_dump())
    except ValidationError as e:
        return None, f"business rule failed: {e.errors()[0]['msg']}"

    # Grounding: values we can check against the source must actually appear in it.
    if invoice.invoice_number not in document_text:
        return None, "invoice number not found in the document"

    return invoice, None

We ran this function against canned responses for each failure case, with the API call replaced by a stub, to check that every path routes where it should:

Write the same kind of test for your own rules. It is cheap, it runs without API calls, and it catches the day someone loosens a validator by accident.

Test on real documents before launch

Stubbed tests prove the routing. They don't prove the model reads your documents well. Before going live, collect a set of real documents with the correct values filled in by a person, run the extractor over them, and compare field by field. Run the same set again whenever you change the prompt, the schema or the model.

Look at two things: how many documents are accepted with a wrong value (the dangerous case), and how many go to review. The first number should be as close to zero as you can get; the second tells you how much human work is left.

Checklist

• Use the provider's structured outputs, not "reply in JSON"
• Keep the model-facing schema plain: types, enums, nesting
• Put business rules in a separate Pydantic model with validators
• Treat any stop reason other than a normal end as a failure
• Check that copyable values (IDs, numbers) appear in the source
• Send every failure to a person, with the reason attached
• Unit-test each routing path with canned responses
• Measure on real documents before launch and after every change

Related service: AI agents and LLM apps →