Agentic AI Cost Optimization: LLM API Budgeting and Token Management

The Cost Problem with Agentic AI in Production

A single agentic AI conversation is surprisingly expensive. The triage agent reads the system prompt (2K tokens), processes the user message, calls the LLM (500 input + 200 output tokens), decides to hand off to a specialist, and passes context. The specialist agent reads its own system prompt (3K tokens), the conversation history (1K tokens), calls a tool, reads the tool result (500 tokens), and generates a response (400 output tokens). That is roughly 7,600 tokens for a simple two-agent interaction.

At Anthropic's Claude Sonnet pricing (USD 3 per million input tokens, USD 15 per million output tokens), that single conversation costs approximately USD 0.03. Multiply by 100,000 conversations per month and you are spending USD 3,000/month — just on a basic agent with minimal tool usage.

Now add multi-turn conversations (5-10 turns each), complex tools that return large payloads, agents that retry on failure, and the cost quickly reaches USD 15,000-50,000 per month for a medium-scale deployment.

At CallSphere, we have reduced our agent LLM costs by over 60% through systematic optimization without sacrificing conversation quality. This guide covers every technique we use.

Understanding Where Tokens Go

Before optimizing, you need to know where your tokens are spent. The typical breakdown for a multi-agent system:

flowchart LR
    subgraph IN["Inputs"]
        I1["Monthly call volume"]
        I2["Average deal value"]
        I3["Current answer rate"]
        I4["Receptionist cost<br/>per month"]
    end
    subgraph CALC["CallSphere Captures"]
        C1["Missed calls converted<br/>at 24 by 7 coverage"]
        C2["Receptionist payroll<br/>displaced or freed"]
    end
    subgraph OUT["Outputs"]
        O1["Recovered revenue<br/>per month"]
        O2["Operating cost saved"]
        O3((Net ROI<br/>monthly))
    end
    I1 --> C1
    I2 --> C1
    I3 --> C1
    I4 --> C2
    C1 --> O1 --> O3
    C2 --> O2 --> O3
    style C1 fill:#4f46e5,stroke:#4338ca,color:#fff
    style C2 fill:#4f46e5,stroke:#4338ca,color:#fff
    style O3 fill:#059669,stroke:#047857,color:#fff

Component	% of Total Tokens	Description
System prompts	25-40%	Repeated on every LLM call
Conversation history	20-30%	Grows with each turn
Tool results	15-25%	Raw data from tools
Agent responses	10-15%	Generated output
Classification/routing	5-10%	Triage decisions

The biggest opportunity is system prompts and conversation history. They are repeated on every single call and grow over time.

Token Counting and Attribution

Implement token counting at every LLM call, attributed to the agent, model, and conversation:

import tiktoken

class TokenTracker:
    def __init__(self):
        self.encoder = tiktoken.get_encoding("cl100k_base")

    def count(self, text: str) -> int:
        return len(self.encoder.encode(text))

    async def track_call(self, agent_name: str, model: str,
                          input_text: str, output_text: str,
                          conversation_id: str):
        input_tokens = self.count(input_text)
        output_tokens = self.count(output_text)
        cost = self.calculate_cost(model, input_tokens, output_tokens)

        await metrics.record({
            "agent": agent_name,
            "model": model,
            "conversation_id": conversation_id,
            "input_tokens": input_tokens,
            "output_tokens": output_tokens,
            "cost_usd": cost,
        })

    def calculate_cost(self, model: str, input_tokens: int, output_tokens: int) -> float:
        rates = MODEL_PRICING.get(model, {"input": 0.003, "output": 0.015})
        return (input_tokens / 1_000_000 * rates["input"]
                + output_tokens / 1_000_000 * rates["output"])

MODEL_PRICING = {
    "claude-3-5-haiku-20241022": {"input": 1.00, "output": 5.00},
    "claude-sonnet-4-20250514": {"input": 3.00, "output": 15.00},
    "claude-opus-4-20250514": {"input": 15.00, "output": 75.00},
    "gpt-4o": {"input": 2.50, "output": 10.00},
    "gpt-4o-mini": {"input": 0.15, "output": 0.60},
}

Technique 1: Prompt Caching

Anthropic's prompt caching stores the compiled representation of your system prompt across calls, so you pay full price only on the first call and a fraction (10% of input token cost) on subsequent calls.

This is the single highest-impact optimization for agentic AI. System prompts are large, static, and repeated on every call — exactly the pattern caching is designed for.

# Without caching: Every call pays full price for the system prompt
# System prompt: 3000 tokens * $3/M = $0.009 per call
# With caching: First call pays full, subsequent pay 10%
# Subsequent calls: 3000 tokens * $0.30/M = $0.0009 per call
# Savings: 90% on system prompt tokens

# Anthropic API with cache_control
response = await client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},  # Enable caching
        }
    ],
    messages=conversation_messages,
)

Cache Optimization Strategy

Structure your prompts so the static portion is at the beginning (and cached) and the dynamic portion is at the end:

# Good: Static prompt cached, dynamic context appended
system_parts = [
    {
        "type": "text",
        "text": STATIC_SYSTEM_PROMPT,  # 3000 tokens - cached
        "cache_control": {"type": "ephemeral"},
    },
    {
        "type": "text",
        "text": TOOL_DEFINITIONS,  # 1500 tokens - cached
        "cache_control": {"type": "ephemeral"},
    },
]
# Dynamic context added as user message, not in system prompt
messages = [
    {"role": "user", "content": f"Context: {dynamic_context}

User: {user_message}"},
]

Technique 2: Model Routing (Cheap for Easy, Expensive for Hard)

Not every agent interaction requires a frontier model. Route simple tasks to cheaper models and reserve expensive models for complex reasoning.

class ModelRouter:
    TIER_MAP = {
        "fast": "claude-3-5-haiku-20241022",     # $1/$5 per M tokens
        "standard": "claude-sonnet-4-20250514",   # $3/$15 per M tokens
        "complex": "claude-opus-4-20250514",      # $15/$75 per M tokens
    }

    TASK_TIERS = {
        "intent_classification": "fast",
        "entity_extraction": "fast",
        "simple_qa": "fast",
        "conversation_routing": "fast",
        "customer_support": "standard",
        "document_analysis": "standard",
        "multi_step_reasoning": "complex",
        "code_generation": "complex",
        "financial_analysis": "complex",
    }

    def select_model(self, task_type: str, conversation_complexity: str = "normal") -> str:
        base_tier = self.TASK_TIERS.get(task_type, "standard")

        # Escalate if conversation is flagged as complex
        if conversation_complexity == "high" and base_tier == "fast":
            base_tier = "standard"

        return self.TIER_MAP[base_tier]

    async def route_with_fallback(self, task_type: str, messages: list) -> dict:
        """Try cheap model first, escalate if response quality is low."""
        model = self.select_model(task_type)
        response = await llm_client.complete(model=model, messages=messages)

        # Check if the response seems inadequate
        if self.needs_escalation(response, task_type):
            better_model = self.escalate_model(model)
            if better_model != model:
                response = await llm_client.complete(model=better_model, messages=messages)

        return response

    def needs_escalation(self, response, task_type: str) -> bool:
        # Heuristics: response too short, contains "I'm not sure",
        # or confidence markers are low
        if len(response.content) < 50 and task_type not in ["intent_classification"]:
            return True
        uncertainty_phrases = ["i'm not sure", "i don't know", "it's unclear"]
        if any(phrase in response.content.lower() for phrase in uncertainty_phrases):
            return True
        return False

Cost Impact of Model Routing

Scenario	Model	Monthly Tokens	Monthly Cost
All conversations on Sonnet	claude-sonnet-4-20250514	500M	$4,500
Routing: 60% Haiku, 30% Sonnet, 10% Opus	Mixed	500M	$2,100
Savings			$2,400 (53%)

Technique 3: Conversation History Management

As conversations grow, the full message history is sent with every LLM call. A 20-turn conversation with tool results can easily reach 10,000+ tokens of history.

Sliding Window with Summarization

class ConversationManager:
    MAX_HISTORY_TOKENS = 4000
    SUMMARY_THRESHOLD = 3000

    def __init__(self, token_counter, summarizer):
        self.counter = token_counter
        self.summarizer = summarizer

    async def prepare_messages(self, full_history: list) -> list:
        """Prepare message history that fits within token budget."""
        total_tokens = sum(self.counter.count(m["content"]) for m in full_history)

        if total_tokens <= self.MAX_HISTORY_TOKENS:
            return full_history

        # Summarize older messages, keep recent ones
        recent_messages = []
        recent_tokens = 0

        for msg in reversed(full_history):
            msg_tokens = self.counter.count(msg["content"])
            if recent_tokens + msg_tokens > self.SUMMARY_THRESHOLD:
                break
            recent_messages.insert(0, msg)
            recent_tokens += msg_tokens

        # Summarize everything before the recent window
        older_messages = full_history[:len(full_history) - len(recent_messages)]
        if older_messages:
            summary = await self.summarizer.summarize(older_messages)
            return [
                {"role": "system", "content": f"Previous conversation summary: {summary}"},
                *recent_messages,
            ]

        return recent_messages

Tool Result Truncation

Tool results are often the largest token consumers. A database query might return 50 rows when the agent only needs the top 3. A web search might return full page content when a snippet suffices.

class ToolResultOptimizer:
    MAX_TOOL_RESULT_TOKENS = 1000

    def truncate_result(self, tool_name: str, result: dict) -> dict:
        """Truncate tool results to reduce token consumption."""
        result_str = json.dumps(result)
        tokens = token_counter.count(result_str)

        if tokens <= self.MAX_TOOL_RESULT_TOKENS:
            return result

        # Strategy 1: If it is a list, take first N items
        if isinstance(result, list):
            truncated = result[:5]
            return {
                "items": truncated,
                "total_count": len(result),
                "truncated": True,
                "message": f"Showing 5 of {len(result)} results",
            }

        # Strategy 2: If it is a dict with large text fields, truncate them
        if isinstance(result, dict):
            truncated = {}
            for key, value in result.items():
                if isinstance(value, str) and len(value) > 500:
                    truncated[key] = value[:500] + "... (truncated)"
                else:
                    truncated[key] = value
            return truncated

        return result

Technique 4: Batch Processing

When processing multiple items (e.g., classifying 100 support tickets), do not make 100 separate LLM calls. Batch them into a single call.

async def batch_classify(items: list, batch_size: int = 10) -> list:
    results = []
    for i in range(0, len(items), batch_size):
        batch = items[i:i + batch_size]
        batch_prompt = "Classify each item below. Return a JSON array.

"
        for j, item in enumerate(batch):
            batch_prompt += f"Item {j+1}: {item['text']}
"

        response = await llm_client.complete(
            model="claude-3-5-haiku-20241022",
            messages=[{"role": "user", "content": batch_prompt}],
        )
        batch_results = json.loads(response.content)
        results.extend(batch_results)

    return results

# 100 items in 10 batches = 10 LLM calls instead of 100
# Token savings: ~80% (shared prompt overhead amortized)

Technique 5: Cost Monitoring and Budget Alerts

Real-Time Cost Dashboard

class CostMonitor:
    def __init__(self, redis_client):
        self.redis = redis_client

    async def record_cost(self, tenant_id: str, agent_name: str, cost_usd: float):
        now = datetime.utcnow()
        hour_key = now.strftime("%Y-%m-%d-%H")
        day_key = now.strftime("%Y-%m-%d")
        month_key = now.strftime("%Y-%m")

        pipe = self.redis.pipeline()
        pipe.incrbyfloat(f"cost:{tenant_id}:hour:{hour_key}", cost_usd)
        pipe.incrbyfloat(f"cost:{tenant_id}:day:{day_key}", cost_usd)
        pipe.incrbyfloat(f"cost:{tenant_id}:month:{month_key}", cost_usd)
        pipe.incrbyfloat(f"cost:agent:{agent_name}:day:{day_key}", cost_usd)
        pipe.expire(f"cost:{tenant_id}:hour:{hour_key}", 172800)
        pipe.expire(f"cost:{tenant_id}:day:{day_key}", 2592000)
        await pipe.execute()

    async def check_budget(self, tenant_id: str) -> dict:
        month_key = datetime.utcnow().strftime("%Y-%m")
        current_cost = float(await self.redis.get(
            f"cost:{tenant_id}:month:{month_key}"
        ) or 0)

        budget = await get_tenant_budget(tenant_id)

        return {
            "current_cost": round(current_cost, 2),
            "budget": budget,
            "usage_pct": round(current_cost / budget * 100, 1) if budget else 0,
            "alert": current_cost > budget * 0.8,
            "blocked": current_cost > budget,
        }

Budget Alert Configuration

Alert Level	Trigger	Action
Info	50% of monthly budget consumed	Email notification to admin
Warning	80% of monthly budget consumed	Slack alert, switch to cheaper models
Critical	95% of monthly budget consumed	Page on-call, enable strict rate limiting
Blocked	100% of monthly budget consumed	Block new conversations, allow active ones to complete

Comprehensive Cost Optimization Impact

Here is the combined impact of all techniques applied to a real deployment processing 100,000 conversations per month:

Technique	Before (Monthly)	After (Monthly)	Savings
Prompt caching	$1,500	$300	80%
Model routing	$3,000	$1,200	60%
History management	$800	$400	50%
Tool result truncation	$600	$200	67%
Batch processing	$400	$80	80%
Response caching (exact)	$200	$50	75%
Total	$6,500	$2,230	66%

Frequently Asked Questions

What is the most impactful single optimization for reducing agentic AI costs?

Prompt caching, followed by model routing. Prompt caching reduces the cost of system prompts by 90% on cache hits, and system prompts typically account for 25-40% of total token consumption. Model routing delivers the next biggest impact by ensuring expensive models are only used when necessary. Implementing just these two techniques typically reduces costs by 50-60%.

How do I prevent cost overruns from runaway agent behavior?

Implement three layers of protection: (1) per-conversation token budgets that terminate conversations exceeding the limit, (2) per-tenant hourly and monthly cost caps tracked in Redis with real-time enforcement, and (3) anomaly detection that alerts when any single conversation or tenant's cost deviates significantly from the baseline. The conversation-level budget is the most critical since it catches infinite loops immediately.

Does using cheaper models for routing hurt conversation quality?

Not when done correctly. Classification and routing tasks are well-suited to smaller models like Claude Haiku or GPT-4o-mini. They can correctly identify user intent over 95% of the time. For the remaining 5% where the fast model is uncertain, escalate to a more capable model. This two-stage approach costs far less than running everything on a frontier model.

How do I estimate costs for a new agent deployment before going to production?

Run 500-1000 representative test conversations through the full agent pipeline in a staging environment. Track token consumption per conversation turn, per agent, and per model. Calculate the average cost per conversation and multiply by your projected monthly volume. Add a 30% buffer for edge cases and multi-turn conversations that are longer than your test set. This estimate is typically accurate within 20% of actual production costs.

Should I self-host an open-source model to reduce costs?

Self-hosting makes economic sense when you process more than 10 million tokens per day of a single task type (like classification) that a smaller open-source model can handle well. Below that volume, the infrastructure costs of GPU instances, model serving, and operational overhead exceed the API savings. A common hybrid approach is to self-host a small model for high-volume, simple tasks (classification, entity extraction) and use API providers for complex reasoning.