An End-to-End Claude Agent Build: Problem to Shipped

Most agent write-ups stop at the demo. They show a clever prompt, a screenshot of Claude doing something impressive, and then nothing about the unglamorous middle where real systems live. The Anthropic Economic Index keeps reminding us that the tasks AI is actually absorbing are concrete operational work — and concrete work needs an end-to-end path, not a magic moment. So let's build one all the way through.

The scenario is deliberately ordinary: a mid-size company drowning in inbound "where is my order?" and refund emails. We'll take it from the messy problem statement to a Claude agent running in production with tools, evals, and guardrails. No hand-waving over the parts that are hard.

Key takeaways

• Start from a measurable problem, not a model — define the outcome and the ground truth before writing a prompt.
• Decompose the task into verifiable units: classify, retrieve, decide, draft, act.
• Wire real tools through MCP so the agent reads orders and drafts replies instead of hallucinating them.
• Build a small eval set early; it is the difference between "seems fine" and "safe to ship."
• Gate the one irreversible action (the refund) behind a check; let everything reversible run autonomously.
• Ship narrow, watch the eval and escalation signals, then widen scope deliberately.

Step one: define the outcome, not the agent

The problem statement "use AI for support" is useless. The shippable version is: cut median first-response time on order-status and refund emails from 6 hours to under 10 minutes, with refund decisions matching policy at least 98% of the time. Now we have a target and, crucially, a definition of correct. The 98% policy-match number is our ground truth; everything downstream serves it.

This is where many builds go wrong — they jump to prompting before anyone has written down what a correct outcome is. Without that, you cannot evaluate the agent, which means you cannot safely automate anything. The Index's automation-versus-augmentation framing helps here: order-status lookups are safe to automate; refund decisions sit closer to augment-and-verify until the eval earns trust.

Step two: decompose into verifiable units

Break the job into stages where each stage has a checkable output. For our support agent: classify the email, retrieve the order, decide the resolution, draft the reply, and (only if needed) execute a refund. Each arrow below is a place we can inspect and test independently.

flowchart TD
  A["Inbound email"] --> B["Classify:\nstatus / refund / other"]
  B --> C["Retrieve order\nvia MCP"]
  C --> D{"Refund\nrequested?"}
  D -->|No| E["Draft status reply"]
  D -->|Yes| F["Decide vs policy"]
  F --> G{"Within policy\n& < $100?"}
  G -->|Yes| H["Issue refund\n(gated) + reply"]
  G -->|No| I["Escalate to human"]

This decomposition is the whole game. Because each unit produces a discrete, inspectable artifact — a classification label, an order record, a policy decision — we can build evals per unit and find the weak link instead of debugging one giant prompt. It also lets us automate the cheap, reversible units while gating the one expensive, irreversible one.

Step three: give the agent real tools via MCP

An agent that guesses order status is worse than useless. We connect Claude to the order system through a Model Context Protocol server so it reads real data. Model Context Protocol is an open standard that lets Claude call external tools and data sources through a consistent interface, so the same agent can talk to your order database, your refund API, and your email drafts without bespoke glue for each.

{
  "name": "get_order",
  "description": "Look up an order by ID or customer email",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_id": { "type": "string" },
      "email": { "type": "string", "format": "email" }
    },
    "required": []
  }
}

That tool definition is the contract: Claude can ask for an order by id or email and gets structured data back. We deliberately give this agent read access to orders and write access to draft replies — but the refund API is a separate, gated tool. Scoping at this layer means a confused or manipulated agent cannot issue money on its own.

Step four: build the eval before you trust it

Pull 50 real historical emails, hand-label the correct classification and the correct refund decision for each, and run the agent against them. This eval set is cheap and decisive. It tells you, before any customer is touched, whether the classifier confuses refunds with returns and whether the policy logic matches your 98% bar.

Each row is a gate. If the refund-decision unit sits at 92%, you do not ship autonomous refunds — you keep that unit in augment-and-verify mode while the others go live. This per-unit shipping is only possible because we decomposed the task in step two.

Step five: ship narrow, then widen

Launch with the smallest safe scope: auto-handle order-status emails fully, draft refund replies for human approval, escalate anything outside policy. Watch two signals daily — eval pass rate on sampled live traffic and the human escalation/override rate. When refunds under $100 hold above 98% policy-match for two weeks, promote that unit from gated to autonomous. Widen by evidence, never by optimism.

Common pitfalls in an end-to-end build

• Prompting before defining correct. Without ground truth you cannot evaluate, and without evaluation you cannot safely automate. Write the outcome and the policy first.
• One mega-prompt instead of units. A single giant prompt is impossible to debug. Decompose so each stage has a checkable artifact.
• Letting the agent hallucinate data. If order status comes from the model's imagination instead of an MCP tool, you've built a liability. Always retrieve real data.
• Skipping the eval set. "It looked good in testing" is how bad refunds reach customers. Fifty labeled examples beat a thousand vibes.
• Shipping wide on day one. Big-bang launches hide which unit is failing. Go narrow, instrument, and expand on evidence.

Ship your first agent in six steps

1. Write the outcome metric and the definition of a correct decision (your ground truth).
2. Decompose the task into units that each produce a checkable artifact.
3. Wire real tools via MCP; scope read, draft, and irreversible actions separately.
4. Hand-label 50 real examples and build a per-unit eval with ship thresholds.
5. Gate the irreversible action; let reversible units run autonomously.
6. Launch narrow, monitor eval and override rates, and widen scope only when the data earns it.

Frequently asked questions

How big should my first eval set be?

Smaller than you think. Fifty carefully hand-labeled, representative examples will catch most systematic failures and give you a defensible ship threshold. Grow it over time by adding every production failure you find, so the eval encodes your actual edge cases rather than hypothetical ones.

Do I need multi-agent for a build like this?

Usually not. A single Claude agent with well-scoped tools handles this support workflow cleanly. Multi-agent designs cost several times more tokens and add coordination risk; reach for them only when a task genuinely needs parallel specialists, not for a linear pipeline like classify-retrieve-decide-draft.

Why connect tools through MCP instead of custom code?

MCP gives you a consistent interface so the same agent can reach orders, refunds, and email without bespoke integration glue per system, and so you can scope each tool's permissions cleanly. It also makes the agent portable across Claude surfaces and easier to audit, since every external action goes through a declared tool.

When is it safe to remove the human gate on refunds?

When the refund-decision unit holds above your policy-match threshold on sampled live traffic for a sustained window — two weeks is a reasonable default — and override rates stay low. Promote one unit at a time, keep the audit log, and be ready to re-gate if the signal slips.

Bringing agentic AI to your phone lines

CallSphere takes this same problem-to-shipped path onto voice and chat: agents that classify a call, pull the right record mid-conversation, and book or resolve work end to end, with humans gating only what matters. Watch a live build at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.