Claude's 200K Context Window: Working Effectively with Long Contexts

Understanding the 200K Context Window

Claude supports a 200,000-token context window -- roughly equivalent to 150,000 words, or a 500-page book. This is one of the largest context windows available among frontier models and fundamentally changes how you can build AI applications.

Instead of complex retrieval-augmented generation (RAG) pipelines that chunk, embed, search, and retrieve document fragments, you can often just put the entire document (or even multiple documents) directly into the prompt. Claude can then answer questions, summarize, compare, and analyze the full content with complete context.

But using a large context window effectively is not as simple as dumping text into a prompt. There are strategies that dramatically improve accuracy, and mistakes that waste tokens without improving results.

The "Lost in the Middle" Problem

Research has shown that LLMs tend to pay more attention to information at the beginning and end of their context, with reduced recall for information in the middle. Claude handles this better than most models -- Anthropic's internal benchmarks show near-flat recall across the full 200K window -- but the effect still exists at the margins.

flowchart LR
    Q(["User query"])
    EMB["Embed query<br/>text-embedding-3"]
    VEC[("Vector DB<br/>pgvector or Pinecone")]
    RET["Top-k retrieval<br/>k = 8"]
    PROMPT["Augmented prompt<br/>system plus context"]
    LLM["LLM generation<br/>Claude or GPT"]
    CITE["Inline citations<br/>and page anchors"]
    OUT(["Grounded answer"])
    Q --> EMB --> VEC --> RET --> PROMPT --> LLM --> CITE --> OUT
    style EMB fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style VEC fill:#ede9fe,stroke:#7c3aed,color:#1e1b4b
    style LLM fill:#4f46e5,stroke:#4338ca,color:#fff
    style OUT fill:#059669,stroke:#047857,color:#fff

Mitigation Strategies

Strategy 1: Put the most important content first and last.

def structure_long_context(documents: list[str], query: str) -> str:
    """Order documents by relevance, placing most relevant at edges."""
    # Score relevance (simple example -- use embeddings in production)
    scored = [(doc, score_relevance(doc, query)) for doc in documents]
    scored.sort(key=lambda x: x[1], reverse=True)

    # Place highest relevance at beginning and end
    n = len(scored)
    ordered = []
    for i, (doc, score) in enumerate(scored):
        if i % 2 == 0:
            ordered.insert(0, doc)  # Add to beginning
        else:
            ordered.append(doc)     # Add to end

    return "\n\n---\n\n".join(ordered)

Strategy 2: Use XML tags to create clear section boundaries.

Claude is specifically trained to attend to XML tags within long contexts. Wrapping sections in descriptive tags significantly improves retrieval:

def format_documents_with_tags(documents: list[dict]) -> str:
    formatted = []
    for i, doc in enumerate(documents):
        formatted.append(f"""<document index="{i+1}" title="{doc['title']}" date="{doc['date']}">
{doc['content']}
</document>""")
    return "\n\n".join(formatted)

Strategy 3: Include explicit retrieval instructions.

system_prompt = """When answering questions about the provided documents:
1. First identify which specific document(s) contain relevant information
2. Quote the exact passage that supports your answer
3. Cite the document by its index number
4. If no document contains the answer, say so explicitly"""

When to Use Long Context vs. RAG

The choice between long context and RAG depends on your specific requirements:

Factor	Long Context (200K)	RAG
Document size	Up to ~500 pages	Unlimited
Accuracy on specific facts	Very high (full context available)	Depends on retrieval quality
Setup complexity	Low (just include documents)	High (embedding, indexing, retrieval)
Latency	Higher TTFT with large contexts	Lower TTFT (smaller prompts)
Cost per query	Higher (processing all tokens)	Lower (only relevant chunks)
Cross-document reasoning	Excellent (all docs in context)	Poor (chunks lack full context)
Maintenance	None (no index to maintain)	Ongoing (re-embed on changes)

The Hybrid Approach

For many applications, the best strategy is a hybrid: use RAG to select the most relevant 50-100K tokens from a larger corpus, then use Claude's long context to process them all together.

async def hybrid_rag_query(query: str, corpus: list[dict]) -> str:
    # Step 1: Use embeddings to find top-K relevant documents
    relevant_docs = await embedding_search(query, corpus, top_k=20)

    # Step 2: Check if they fit in context (leave room for system + output)
    total_tokens = sum(count_tokens(doc["content"]) for doc in relevant_docs)

    while total_tokens > 150_000:  # Leave 50K for system prompt + output
        relevant_docs.pop()  # Remove least relevant
        total_tokens = sum(count_tokens(doc["content"]) for doc in relevant_docs)

    # Step 3: Send all relevant docs to Claude in a single call
    context = format_documents_with_tags(relevant_docs)

    response = client.messages.create(
        model="claude-sonnet-4-5-20250514",
        max_tokens=4096,
        system=[{
            "type": "text",
            "text": "You are a research assistant. Answer based on the provided documents.",
            "cache_control": {"type": "ephemeral"},
        }],
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": context, "cache_control": {"type": "ephemeral"}},
                {"type": "text", "text": query},
            ]
        }],
    )
    return response.content[0].text

Cost Management with Long Contexts

Processing 200K tokens is not cheap. At Claude Sonnet rates ($3/M input), a full context window costs $0.60 per request. For multi-turn conversations where context accumulates, costs compound.

Strategies to Control Costs

1. Trim conversation history aggressively.

def trim_conversation(messages: list[dict], max_tokens: int = 100_000) -> list[dict]:
    """Keep the system prompt and most recent messages within budget."""
    total = 0
    trimmed = []

    # Always keep the most recent messages (iterate in reverse)
    for msg in reversed(messages):
        msg_tokens = count_tokens(str(msg))
        if total + msg_tokens > max_tokens:
            break
        trimmed.insert(0, msg)
        total += msg_tokens

    return trimmed

2. Summarize older context.

Instead of keeping all raw conversation history, periodically summarize older turns:

async def compress_history(messages: list[dict]) -> str:
    """Use Haiku to summarize older conversation turns."""
    old_messages = messages[:-6]  # Keep last 3 exchanges raw

    response = client.messages.create(
        model="claude-haiku-4-5-20250514",  # Use cheapest model for summarization
        max_tokens=1024,
        system="Summarize this conversation, preserving all key facts and decisions.",
        messages=[{"role": "user", "content": format_messages(old_messages)}]
    )
    return response.content[0].text

3. Use prompt caching.

For contexts that do not change between turns (system prompts, reference documents), prompt caching reduces cost by 90% on cached portions.

Practical Examples

Entire Codebase Analysis

import os

def collect_codebase(directory: str, extensions: set = {".py", ".ts", ".js"}) -> str:
    files = []
    for root, dirs, filenames in os.walk(directory):
        dirs[:] = [d for d in dirs if d not in {"node_modules", ".git", "__pycache__", "venv"}]
        for fname in filenames:
            if any(fname.endswith(ext) for ext in extensions):
                filepath = os.path.join(root, fname)
                with open(filepath) as f:
                    content = f.read()
                files.append(f"<file path=\"{filepath}\">\n{content}\n</file>")

    return "\n\n".join(files)

codebase = collect_codebase("./src")
# Now send to Claude for analysis, refactoring suggestions, bug hunting, etc.

Multi-Document Legal Review

contracts = load_contracts(["vendor_a.pdf", "vendor_b.pdf", "vendor_c.pdf"])
formatted = format_documents_with_tags(contracts)

response = client.messages.create(
    model="claude-sonnet-4-5-20250514",
    max_tokens=8192,
    system="You are a contract analyst. Compare these contracts and identify key differences.",
    messages=[{
        "role": "user",
        "content": f"""{formatted}

Compare these three vendor contracts. For each of the following areas,
create a comparison table showing the terms from each vendor:
1. Pricing and payment terms
2. Liability and indemnification
3. Termination clauses
4. SLA commitments
5. Data handling and privacy"""
    }]
)

Performance Tips

• Pre-count tokens before sending requests. Use Anthropic's tokenizer or approximate at 4 characters per token
• Set appropriate max_tokens for output -- do not request 4,096 output tokens if you only need a short answer
• Use streaming for long-context requests to get faster time to first token
• Batch similar queries against the same context to amortize the input cost across multiple questions