Output Formatting: Getting Structured JSON, Markdown, and CSV from LLMs

The Structured Output Challenge

LLMs generate free-form text by default. But production applications need structured data — JSON objects to store in databases, CSV rows for spreadsheets, or markdown with specific formatting. Getting reliable structured output requires a combination of prompt design, API features, and defensive parsing.

JSON Mode: The Built-In Solution

Most modern LLM APIs offer a JSON mode that constrains the model to produce valid JSON:

flowchart TD
    SPEC(["Task spec"])
    SYSTEM["System prompt<br/>role plus rules"]
    SHOTS["Few shot examples<br/>3 to 5"]
    VARS["Variable injection<br/>Jinja or f-string"]
    COT["Chain of thought<br/>or scratchpad"]
    CONSTR["Output constraint<br/>JSON schema"]
    LLM["LLM call"]
    EVAL["Offline eval<br/>LLM as judge plus regex"]
    GATE{"Score over<br/>threshold?"}
    COMMIT(["Promote to prod<br/>version pinned"])
    REVISE(["Revise prompt"])
    SPEC --> SYSTEM --> SHOTS --> VARS --> COT --> CONSTR --> LLM --> EVAL --> GATE
    GATE -->|Yes| COMMIT
    GATE -->|No| REVISE --> SYSTEM
    style LLM fill:#4f46e5,stroke:#4338ca,color:#fff
    style EVAL fill:#f59e0b,stroke:#d97706,color:#1f2937
    style COMMIT fill:#059669,stroke:#047857,color:#fff

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},
    messages=[
        {
            "role": "system",
            "content": "Extract contact information from the text. Return a JSON object with fields: name, email, phone, company. Use null for missing fields."
        },
        {
            "role": "user",
            "content": "Hi, I'm Sarah Chen from Acme Corp. Reach me at sarah@acme.com or 555-0142."
        }
    ]
)

import json
data = json.loads(response.choices[0].message.content)
print(data)
# {"name": "Sarah Chen", "email": "sarah@acme.com", "phone": "555-0142", "company": "Acme Corp"}

JSON mode guarantees syntactically valid JSON but does not guarantee the schema matches your expectations. You still need validation.

Schema Enforcement with Pydantic

Combine JSON mode with Pydantic for type-safe structured outputs:

from pydantic import BaseModel, Field
from openai import OpenAI
import json

class ContactInfo(BaseModel):
    name: str = Field(description="Full name of the person")
    email: str | None = Field(default=None, description="Email address")
    phone: str | None = Field(default=None, description="Phone number")
    company: str | None = Field(default=None, description="Company name")
    role: str | None = Field(default=None, description="Job title or role")

def extract_contact(text: str) -> ContactInfo:
    client = OpenAI()

    schema_description = json.dumps(
        ContactInfo.model_json_schema(), indent=2
    )

    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[
            {
                "role": "system",
                "content": f"Extract contact information from the text. "
                           f"Return JSON matching this schema:\n{schema_description}"
            },
            {"role": "user", "content": text}
        ]
    )

    raw = json.loads(response.choices[0].message.content)
    return ContactInfo.model_validate(raw)

contact = extract_contact("John Doe, CTO at TechStartup Inc — john@techstartup.io")
print(contact.name)     # "John Doe"
print(contact.role)     # "CTO"
print(contact.company)  # "TechStartup Inc"

Pydantic validates every field type, applies defaults for missing fields, and raises clear errors when the model returns unexpected data.

Handling Malformed Responses

Even with JSON mode, things go wrong in production. Build defensive parsing:

import json
import re
from typing import TypeVar
from pydantic import BaseModel, ValidationError

T = TypeVar("T", bound=BaseModel)

def safe_parse(response_text: str, model_class: type[T]) -> T | None:
    """Parse LLM response into a Pydantic model with fallback strategies."""
    # Strategy 1: Direct JSON parse
    try:
        data = json.loads(response_text)
        return model_class.model_validate(data)
    except (json.JSONDecodeError, ValidationError):
        pass

    # Strategy 2: Extract JSON from markdown code blocks
    json_match = re.search(r'```(?:json)?s*
(.*?)
```', response_text, re.DOTALL)
    if json_match:
        try:
            data = json.loads(json_match.group(1))
            return model_class.model_validate(data)
        except (json.JSONDecodeError, ValidationError):
            pass

    # Strategy 3: Find the first { ... } block
    brace_match = re.search(r'{.*}', response_text, re.DOTALL)
    if brace_match:
        try:
            data = json.loads(brace_match.group(0))
            return model_class.model_validate(data)
        except (json.JSONDecodeError, ValidationError):
            pass

    return None

This layered approach handles the three most common failure modes: raw JSON, JSON wrapped in markdown, and JSON embedded in explanatory text.

Structured Markdown Output

For human-readable outputs, constrain the markdown format explicitly:

markdown_prompt = """Analyze the provided server logs and produce a report in this exact format:

# Incident Report

## Summary
(One paragraph describing the incident)

## Timeline
| Time | Event | Severity |
|------|-------|----------|
(Fill in rows from the log data)

## Root Cause
(One paragraph identifying the root cause)

## Remediation Steps
1. (Numbered steps to fix the issue)

## Prevention
- (Bullet points for preventing recurrence)

Do not add any sections not listed above. Do not change the heading names."""

CSV Output for Data Pipelines

When you need tabular data, CSV format with explicit column definitions works reliably:

csv_prompt = """Extract all product mentions from the review text.

Return a CSV with these exact columns (include the header row):
product_name,sentiment,confidence,evidence

Rules:
- sentiment must be one of: positive, negative, neutral
- confidence must be a float between 0.0 and 1.0
- evidence is the exact quote from the review (in double quotes if it contains commas)
- One row per product mention
- No extra text before or after the CSV"""

import csv
import io

def parse_csv_response(response_text: str) -> list[dict]:
    """Parse CSV from LLM response into a list of dicts."""
    # Strip any leading/trailing whitespace or markdown fences
    cleaned = response_text.strip()
    if cleaned.startswith("```"):
        cleaned = re.sub(r'```(?:csv)?s*
?', '', cleaned).strip()

    reader = csv.DictReader(io.StringIO(cleaned))
    return [row for row in reader]

FAQ

Should I always use JSON mode?

Use JSON mode whenever you need to parse the response programmatically. For human-readable outputs (reports, summaries, explanations), plain text or markdown is more appropriate. JSON mode adds a small amount of latency due to the constrained decoding, so avoid it when you just need natural language.

What do I do when the model ignores my schema?

First, simplify the schema — models struggle with deeply nested objects (3+ levels). Second, provide a concrete example of the expected output in your prompt. Third, validate and retry: if parsing fails, send the error message back to the model and ask it to fix the output.

Can I get the model to produce valid CSV reliably?

CSV is less reliable than JSON because models frequently introduce formatting issues (missing quotes, extra commas, inconsistent escaping). For critical data pipelines, prefer JSON output and convert to CSV in your code. If you must use CSV, always validate the row count and column count before processing.

---

#StructuredOutput #JSONMode #PromptEngineering #DataExtraction #Python #AgenticAI #LearnAI #AIEngineering