Claude Agent SDK Walkthrough: Problem to Shipped

Most agent tutorials stop at a working demo. The interesting part is everything after the demo: the day you connect the agent to real data, real users, and real consequences, and discover which of your assumptions survive contact with production. This post walks through one realistic build end to end — not a toy, but the kind of project a small team actually ships in a few weeks with the Claude Agent SDK. The shape generalizes even if your domain differs.

The problem: a mid-sized software company has a support queue that's drowning. Tier-one tickets pile up overnight, most of them are variations on a few dozen known issues, and customers wait hours for a reply that a knowledgeable human could write in two minutes. Leadership wants faster responses without hiring a night shift. Classic agent territory — repetitive, knowledge-driven, high-volume — but also customer-facing, which means getting it wrong is visible and costly.

Scoping the problem before writing code

The first decision was deliberately narrow. We did not build an agent to "handle support." We built one to draft replies for a specific category of tickets — billing questions — and route everything else untouched to humans. Narrow scope is the single highest-leverage choice in an agent project. It shrinks the failure surface, makes success measurable, and gives you something shippable in weeks instead of quarters.

We also decided up front that the agent would draft, not send. For the first phase a human reviewed every reply before it reached a customer. That single constraint took the scariest risk — a confidently wrong message going out — off the table while we learned how the agent actually behaved on real tickets. You can always remove a guardrail later once the data earns it.

Designing the tools and the loop

With scope fixed, the build came down to three things: the tools the agent could use, the context it received, and the loop that tied them together.

flowchart TD
  A["New billing ticket"] --> B["Agent reads ticket"]
  B --> C["search_docs tool"]
  C --> D["lookup_account tool"]
  D --> E{"Enough to answer?"}
  E -->|No| F["Flag for human"]
  E -->|Yes| G["Draft reply"]
  G --> H["Human review"]
  H --> I["Send & log outcome"]

We exposed exactly three tools through MCP. search_docs queried the help center and billing policies. lookup_account fetched a customer's plan and recent invoices, read-only and scoped to the requesting ticket. flag_for_human let the agent bail out cleanly whenever it wasn't confident. That last tool mattered more than the other two — an agent that knows when to stop is worth far more than one that always answers. We wrote each tool description as if the model were a new hire reading it cold, because effectively it was.

Context engineering was the other half. Rather than dumping the entire knowledge base into the prompt, we let the agent pull only the docs relevant to the ticket via Agent Skills and the search tool. The account data came in as structured fields, not raw JSON dumps, so the model spent its attention reasoning rather than parsing. Keeping the context lean kept both quality and cost in check.

Building the eval harness first

Before letting the agent touch a live ticket, we built an eval set: roughly a hundred real historical billing tickets with known good resolutions. For each, we scored the agent's draft on whether it was factually correct, whether it used the right tools, and whether it correctly flagged the ones a human should handle. This harness became the project's backbone. Every prompt change, every tool tweak, every model swap ran against it before going anywhere near production. Without it, we'd have been tuning on vibes; with it, we could see a regression the moment it appeared.

The first eval run was humbling. The agent answered confidently on tickets it should have flagged, and it occasionally cited a policy that didn't apply to the customer's plan. Both were fixable — the first with a sharper flagging instruction and a lower confidence threshold, the second by passing the customer's plan tier into context so the agent stopped reaching for the wrong policy. Each fix was validated on the eval set, not assumed.

Shipping behind a gate, then widening

We shipped to production with the human-review gate firmly in place and a hard cap on how many tickets the agent would touch per hour. For the first week we watched everything: every draft, every flag, every tool call. Reviewers corrected drafts and those corrections fed straight back into the eval set, so the agent's test bar rose as we learned. Acceptance — the share of drafts a human sent with little or no edit — climbed steadily as the rough edges got filed down.

Only once acceptance was consistently high, and the flagging behavior was reliably conservative, did we discuss loosening the gate. Even then we widened it narrowly: auto-send for a small, well-understood subset of billing questions where the agent had a strong track record, with humans still reviewing everything else. That is the right tempo for agent rollouts — earn each increment of autonomy with evidence, and never grant more trust than the data supports.

What the build taught us

Three lessons generalized. First, narrow scope plus a human gate let us ship fast and safely; the agent was useful in week three because we hadn't tried to boil the ocean. Second, the eval harness was the most valuable artifact we built — more than any prompt. Third, the bail-out tool that let the agent decline was what made customers trust the system, because a flagged ticket reaching a human beats a wrong answer reaching a customer every time.

Frequently asked questions

How long does a first production agent take to build?

A narrowly scoped agent like this one is typically a few weeks, not months. The model and SDK aren't the bottleneck — tool design, the eval harness, and the production gating take most of the time, and a tight scope keeps all three small.

Should an agent send customer messages automatically at first?

No. Start with draft-and-review so a human catches mistakes while you learn the agent's behavior. Widen to auto-send only for the specific subset where your evals and production track record justify it.

Why build the eval set before shipping?

Because without it you're tuning blind. An eval set of real historical cases with known good outcomes lets you catch regressions instantly and validate every change, instead of discovering problems in front of customers.

What made the agent trustworthy to the team?

The bail-out tool. An agent that reliably flags what it can't handle, rather than guessing, earns trust quickly — a correctly escalated ticket is always better than a confidently wrong reply.

Bringing agentic AI to your phone lines

This same problem-to-production arc applies to live conversations. CallSphere builds voice and chat agents that answer every call and message, pull data mid-conversation, and escalate cleanly when needed — shipped with the same gates and evals. See it at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.