Cutting AI Agent Cost: Caching, Batching, Fast Runs

A working agent and an affordable agent are two different engineering problems. The version you demo on day one re-sends the same 8,000-token system prompt on every step, runs everything on your most capable model, and processes requests one at a time. It works — and it costs ten times what it should. Cost and latency are not afterthoughts you optimize once; they are design constraints that shape how you structure context, route work, and schedule runs. This post is about the levers that actually move the bill: prompt caching, batching, model routing, and context discipline.

For grounding: prompt caching is a feature that stores the processed form of a stable prefix of your prompt so that repeated requests reusing that prefix skip most of the input-token cost and latency. Combined with batching and smart model selection, it is the difference between an agent that is viable at scale and one that gets switched off when the finance team sees the invoice.

Key takeaways

• Prompt caching is the single highest-leverage cost lever for agents — stable prefixes (system prompt, tools, docs) can be cached so repeat reads are far cheaper and faster.
• Order your context cheapest-to-change-last: static content first, dynamic content last, so the cache prefix stays warm across turns.
• Use the Message Batches API for any non-interactive workload — it trades latency for a large discount on jobs you don't need answered instantly.
• Route by difficulty: send routine steps to a smaller, cheaper model and reserve the most capable model for genuinely hard reasoning.
• The cheapest token is the one you never send — prune tool outputs, summarize long histories, and avoid dumping whole files into context.
• Measure tokens-per-task, not tokens-per-call; a multi-agent run can quietly cost several times a single-agent run.

Where the money actually goes

In an agent loop, the same context gets re-processed on every single turn. If your agent takes twelve tool-call iterations to finish a task, your fixed system prompt and tool definitions are read by the model twelve times. Multiply that across thousands of runs and the static parts of your prompt — not the dynamic user input — are usually the biggest line item. This is exactly the cost prompt caching attacks. By caching the stable prefix, you pay full price to write it to cache once, then a small fraction to read it on every subsequent turn within the cache lifetime.

The implication for your architecture is concrete: structure context so the unchanging parts come first and the changing parts come last. A cache hit requires the prefix to match exactly, so if you interleave dynamic data into your system prompt, you destroy the cache on every turn. Keep system instructions, tool schemas, and reference documents at the top; put the live conversation and fresh data at the bottom.

The optimization decision flow

Not every workload wants the same treatment. The flow below is how I decide which lever to pull for a given agent task.

flowchart TD
  A["Agent task"] --> B{"Needs an answer now?"}
  B -->|No| C["Send via Message Batches API"]
  B -->|Yes| D{"Stable prefix reused?"}
  D -->|Yes| E["Enable prompt caching on prefix"]
  D -->|No| F["Restructure: static first"]
  E --> G{"Step difficulty?"}
  G -->|Routine| H["Route to Haiku / Sonnet"]
  G -->|Hard reasoning| I["Route to Opus"]
  C --> J["Collect results, lower cost"]
  H --> K["Cheaper, faster run"]
  I --> K

Caching in practice

With the Claude API you mark the end of a cacheable prefix using a cache control breakpoint. Everything before the breakpoint becomes the reusable, cached segment. Here is the shape for an agent whose system prompt and tool definitions are stable across the whole session:

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {"type": "text", "text": LONG_STATIC_INSTRUCTIONS},
        {"type": "text",
         "text": REFERENCE_DOCS,
         "cache_control": {"type": "ephemeral"}}  # cache everything up to here
    ],
    tools=TOOLS,                 # also stable -> lands inside the cached prefix
    messages=conversation,       # only this changes per turn
)

The rule of thumb: place the cache breakpoint after the last thing that never changes. On the first call you pay a small write premium; on every subsequent call in the session you read that whole block at a steep discount. For long-running agents with big system prompts, this routinely cuts input cost dramatically and shaves latency because cached tokens don't need full reprocessing.

Batching the work that can wait

A huge share of agent work is not interactive: nightly enrichment, bulk classification, generating summaries for a backlog, running an eval suite. For all of it, the Message Batches API lets you submit many requests as one job and accept results within a window rather than instantly, in exchange for a substantial per-token discount. If a user is not staring at a spinner, you are probably overpaying by running it synchronously. The mental model: synchronous API for anything a human is waiting on, batch API for everything else.

Common pitfalls

• Cache-busting your own prefix. Injecting a timestamp, request ID, or per-user variable near the top of the system prompt invalidates the cache every call. Keep volatile values at the bottom.
• One model for everything. Running trivial routing and parsing steps on your most expensive model wastes money. Route by difficulty.
• Letting tool outputs balloon. Dumping a 50KB API response straight into context inflates every later turn. Summarize or extract only the fields the agent needs.
• Reaching for multi-agent by default. Multi-agent systems can use several times the tokens of a single agent. Use them when parallel exploration genuinely pays, not reflexively.
• Optimizing per-call instead of per-task. A cheaper individual call that triples the number of turns is a regression. Always measure end-to-end tokens per completed task.

Make an agent cheap in 5 steps

1. Measure baseline tokens-per-completed-task and latency before changing anything.
2. Reorder context so static content (instructions, tools, docs) sits first and add a cache breakpoint after it.
3. Profile which steps are routine vs. hard, and route routine steps to a smaller model.
4. Move every non-interactive workload to the Message Batches API.
5. Trim context: cap tool-output size, summarize long histories, and re-measure tokens-per-task.

Cost lever comparison

Lever	Best for	Tradeoff	Typical impact
Prompt caching	Agents reusing a large stable prefix	Small write premium on first call	Large input-cost & latency drop
Message batching	Non-interactive bulk jobs	Results within a window, not instant	Large per-token discount
Model routing	Mixed-difficulty workloads	Routing logic to maintain	Lower cost on routine steps
Context pruning	Long multi-turn runs	Engineering effort to summarize	Compounding savings every turn

Frequently asked questions

Does prompt caching change the model's answers?

No. Caching only reuses the processed form of an identical prefix; the model sees the same tokens and produces the same quality of output. It is a cost and latency optimization, not a behavioral one.

When should I use the Message Batches API instead of the regular API?

Whenever no human is waiting on the result — overnight enrichment, bulk classification, eval runs, backfills. If you can tolerate results arriving within a window rather than instantly, batching saves meaningfully.

How much can I really save by routing to smaller models?

It depends on your task mix, but routine steps — routing, extraction, simple formatting — often run perfectly well on a smaller model at a fraction of the cost, freeing your most capable model for the genuinely hard reasoning.

Why did my multi-agent system get so expensive?

Each subagent carries its own context and tool reads, so a coordinated multi-agent run commonly uses several times the tokens of a single agent. Reserve the pattern for tasks where parallel breadth clearly outweighs the cost.

Bringing agentic AI to your phone lines

CallSphere runs these same cost disciplines — caching stable prompts, batching the non-urgent work, and routing by difficulty — so its voice and chat agents stay fast and affordable while answering every call and message 24/7. See the live system at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.