End-to-end: shipping a Claude agent from problem to prod

Most writing about Claude agents stays at the level of capabilities and patterns. This post does the opposite. It follows one realistic project from a vague business problem to a shipped, measured outcome, including the dead ends and the decisions that actually determined whether it worked. The specifics are composite, but every step is the kind of thing real teams hit when they build an agent across their developer tools.

The problem: a mid-size software company's support team is drowning in inbound tickets about a single product area, integrations. Engineers get pulled off roadmap work to answer the same configuration questions every week. Leadership wants an agent that drafts accurate first responses to integration tickets, pulling from the codebase, the docs, and past resolved tickets. The constraint: it must be accurate enough that support agents trust it, not a chatbot that hallucinates config steps.

Step one: turn the ask into a spec

The first real work is not code. It is converting "build an agent that answers integration tickets" into something precise enough to build and measure. The team writes down what good looks like: a draft response that cites the specific doc or code path it relied on, flags when it is unsure, and never invents a configuration option that does not exist. They also write down what is out of scope: the agent drafts, a human sends. No auto-reply to customers.

This step kills the most common failure before it happens. An agent built against a vague goal will be confidently wrong; an agent built against "cite your source and flag uncertainty" has a fighting chance. The spec becomes the contract the rest of the project is measured against.

Step two: give Claude the right tools and context

The agent needs to reach three systems: the docs, the codebase, and the history of resolved tickets. The team exposes each through an MCP server. One server lets Claude search the documentation. Another lets it search the integration source code. A third queries the resolved-ticket archive for similar past cases. They deliberately do not give the agent write access to anything customer-facing; its only output is a draft into the support tool.

Alongside the tools, they build an Agent Skill that encodes the house rules: how to cite a source, how to phrase uncertainty, the tone support uses, and the specific integrations that have known sharp edges. The skill is what turns a generic capable model into a teammate that answers the way this company answers. Tools give reach; the skill gives judgment.

flowchart TD
  A["New integration ticket"] --> B["Claude agent reads ticket"]
  B --> C{"Find similar resolved case?"}
  C -->|Yes| D["Pull past resolution via MCP"]
  C -->|No| E["Search docs + code via MCP"]
  D --> F["Draft answer with citations"]
  E --> F
  F --> G{"Confidence high?"}
  G -->|Yes| H["Draft to support agent"]
  G -->|No| I["Flag uncertain + escalate"]

Step three: build the eval before trusting the agent

Here is where this team avoids the trap that sinks most agent projects. Before letting the agent touch live tickets, they build an eval set: 60 past tickets with known-good resolutions. They run the agent against all 60 and score each draft on whether it cited a real source, whether the configuration steps were correct, and whether it correctly flagged the cases that genuinely needed a human.

The first run is humbling. The agent scores well on tone and citation but gets configuration steps wrong on a class of tickets involving a legacy integration whose docs are stale. This is exactly the kind of failure the eval exists to catch. The team fixes it not by retraining anything but by updating the Agent Skill to tell the agent that this integration's docs are unreliable and to prefer resolved-ticket history for it. The eval score jumps on the next run.

Step four: ship behind a human, then widen

The agent goes live in the safest possible mode: it drafts into the support tool, and a human support agent reviews and sends every response. Nothing reaches a customer without a person in the loop. The team watches two things closely: how often support agents send the draft with minimal edits, and how often they discard it entirely.

In the first two weeks, support agents send roughly two-thirds of drafts with light edits and discard the rest. That discard rate is the feedback loop. The team reads the discarded drafts, finds the patterns, and feeds the fixes back into the skill and the eval set. The agent gets better not through magic but through this tight cycle of observe, diagnose, adjust, re-measure.

Step five: measure the outcome that leadership asked for

The project did not start because anyone wanted an agent. It started because engineers were being pulled off roadmap work. So the outcome that matters is engineering hours reclaimed and first-response time on integration tickets. After a month, first-response time on that ticket class drops sharply because drafts are ready the moment a support agent opens the ticket, and engineer escalations fall because support resolves more cases without pulling in an engineer.

Crucially, the team can attribute the change because they measured the baseline before shipping. Without that baseline, they would have a working agent and no way to prove it mattered, which is how good agent projects quietly lose their funding.

What made this work

Step back and the pattern is clear. The agent succeeded not because of a clever prompt but because the team specified before building, scoped tools tightly, built an eval before trusting the output, shipped behind a human, and measured against the real business goal. Take away any one of those and the project gets shakier. The model was the easy part. The engineering discipline around the model was the project.

Frequently asked questions

How long does a project like this take?

A focused team can go from spec to a human-in-the-loop pilot in a few weeks, with most of the time spent on the eval set and skill tuning rather than wiring up the model. Widening from pilot to steady state takes longer because it depends on the feedback loop.

Why build the eval before going live?

Because the eval catches the failures that would otherwise reach customers, and because it gives you a number to improve against. Without an eval, you tune by anecdote and never know whether a change helped or hurt overall.

Why keep a human in the loop if the agent is accurate?

The human-in-the-loop mode is what makes shipping safe early and generates the feedback that improves the agent. You can widen autonomy later once the discard rate is low and trust is earned, but starting with a human costs little and prevents the worst failures.

What if the underlying docs are wrong?

Encode that in the Agent Skill, as this team did with their legacy integration. Tell the agent which sources are unreliable and which to prefer instead. The skill is the right place to capture institutional knowledge about where your own data lies.

From shipped agent to shipped conversation

CallSphere takes this same problem-to-production discipline to voice and chat, where agents answer live, cite real data through tools, and book work without a human babysitting every turn. See a worked example at callsphere.ai.

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.