Claude Agent Walkthrough: Problem to Shipped Outcome

Most write-ups about enterprise AI agents stop at the architecture diagram. Real value lives in the messy middle: the part where you take a vague business complaint, turn it into something Claude can actually run, fight the edge cases, prove it works, and put it in front of customers. This post walks that whole path for one realistic use case, end to end, with the decisions and dead-ends an actual team hits.

The scenario: a mid-size SaaS company has a support backlog problem. Tier-1 agents spend most of their day on a narrow set of account questions — billing status, plan changes, usage limits — and the queue is always behind. Leadership wants Claude to handle the repetitive tier-1 work without making things worse. Here is how that goes from problem statement to a shipped, measured agent.

Key takeaways

• Start from a narrow, high-volume, well-bounded slice of work — not "handle all support."
• Spend the first week writing the spec and the eval set, not building; the gold cases define "done."
• Give the agent typed MCP tools against real systems, capped server-side, instead of broad access.
• Ship behind a human-in-the-loop suggestion mode first; promote to autonomous only on the easy intents that pass eval.
• Measure deflection, accuracy, and escalation rate from day one — the numbers decide what to expand next.

Step 1 — Turn the complaint into a scoped problem

"Handle support with AI" is unbuildable. The first job is narrowing. We pull a month of tickets and tag them: it turns out billing-status, plan-change, and usage-limit questions are a large share of tier-1 volume and are highly templated — they map to a few backend lookups and a clear answer. Password resets and outage reports, by contrast, are messy and high-stakes. We scope the agent to exactly the three templated intents and explicitly exclude the rest. That exclusion is a feature: a narrow agent that does three things reliably beats a broad agent that does twenty things unpredictably.

We also write down the blast radius. The agent can read account data and answer questions; the only mutating action we allow at first is initiating a plan change, and that we cap and gate. Everything else is read-only. This scoping conversation is most of the risk work, and it takes an afternoon, not a sprint.

Step 2 — Write the eval set before building

Before a line of agent code, we build a gold set: roughly fifty real (anonymized) tickets across the three intents, each with the correct answer and the correct action. This is the single most important artifact in the project. It defines what "working" means, it catches regressions later, and it tells us when we are done. Teams that skip this build by vibes and have no way to know if a change made things better or worse.

flowchart TD
  A["Inbound ticket"] --> B{"In-scope intent?"}
  B -->|No| C["Escalate to human"]
  B -->|Yes| D["Claude reads account via MCP tools"]
  D --> E{"Confident & low-risk?"}
  E -->|No| C
  E -->|Yes| F["Draft answer / propose action"]
  F --> G{"Mutating action?"}
  G -->|Yes| H["Gate: cap + log"]
  G -->|No| I["Send reply"]
  H --> I
  I --> J["Log outcome to eval + metrics"]

The flow above is what we are building toward. Note the two escape hatches to a human: out-of-scope intent, and low confidence. Those two branches are what keep the agent from confidently mishandling the hard cases it was never meant to touch.

Step 3 — Wire real tools through MCP

The agent needs to see real account data, so we expose the billing and account systems through Model Context Protocol servers with narrow, typed tools. Crucially, the tools are scoped — read-only lookups plus one capped mutation — and the cap is enforced on the server, not in the prompt.

{
  "tools": [
    { "name": "get_account_status",
      "description": "Return plan, billing state, and usage for an account.",
      "input_schema": { "type": "object",
        "properties": { "account_id": { "type": "string" } },
        "required": ["account_id"] } },
    { "name": "change_plan",
      "description": "Change an account's plan. Server rejects downgrades that lose data and any change over the tier limit.",
      "input_schema": { "type": "object",
        "properties": { "account_id": { "type": "string" },
                        "new_plan": { "type": "string",
                          "enum": ["starter","growth","scale"] } },
        "required": ["account_id","new_plan"] } }
  ]
}

The enum on new_plan and the server-side rejection rules mean the agent literally cannot propose an invalid or dangerous change. This is the difference between a demo and something you trust against production data: the safety lives in the tool contract, not in hoping the model behaves.

Step 4 — Ship in suggestion mode, then promote

We do not turn the agent loose on customers on day one. First it runs in suggestion mode: it drafts replies and proposed actions that a human agent reviews and sends. This does two things — it keeps the blast radius at zero while we learn, and it generates a stream of human corrections that become new eval cases. We watch the agreement rate: how often does the human accept the draft unedited?

Once the billing-status intent clears a high bar on the eval set and a strong real-world agreement rate, we promote just that intent to autonomous, still logging everything. Plan-changes stay gated behind human approval longer because they mutate state. Usage-limit answers, being read-only, graduate quickly. This staged promotion — easy read-only intents first, mutating intents last — is how you ship without betting the customer experience on a single launch.

Intent	Risk	First mode	Promote when
Usage-limit Q	Read-only	Suggestion	High eval + agreement
Billing-status Q	Read-only	Suggestion	High eval + agreement
Plan change	Mutating	Gated approval	Sustained accuracy + audit clean

Step 5 — Measure, then expand

From the first suggestion-mode day we track four numbers: deflection (share of in-scope tickets fully resolved without a human), accuracy (graded against the gold set and via spot audits), escalation rate (how often it correctly hands off), and customer satisfaction on agent-handled tickets versus human-handled. These numbers are the feedback loop that decides what to build next. If deflection is high and satisfaction holds, we add the next intent. If accuracy dips after a model upgrade, the eval catches it before customers do.

The shipped outcome is not "AI handles support." It is a narrow agent that reliably resolves three high-volume intents, hands off cleanly on everything else, and has the instrumentation to prove it. That is a real transformation, and it generalizes: every subsequent intent is a smaller version of the same loop — scope, eval, tools, staged rollout, measure.

It is worth being honest about the dead-ends this build hit, because they are typical. The first version of the plan-change prompt let Claude infer the target plan from loose phrasing like "the bigger one," and it occasionally guessed wrong; we fixed it by forcing the agent to confirm the exact plan name before proposing the change. The first eval set was too easy — all clean, well-formed tickets — so the agent looked perfect in testing and stumbled on real messy phrasing; we rebuilt the gold set from actual production tickets, typos and all. And we initially under-escalated, because the agent was a little too eager to answer; tightening the confidence threshold raised escalation slightly but cut wrong answers sharply, which the metrics confirmed was the right trade. None of these were model failures. They were specification failures, and each one became a permanent eval case so it could never silently return.

Common pitfalls

• Scoping too broad. "Handle all support" never ships; pick three intents and exclude the rest explicitly.
• Building before evals. Without a gold set you cannot tell improvement from regression, and you ship on vibes.
• Broad tool access. Raw API keys or SQL make the blast radius huge; use typed, capped MCP tools.
• Going autonomous on day one. Suggestion mode first generates corrections and keeps risk at zero while you learn.
• No post-launch metrics. Without deflection and accuracy tracking, you cannot defend the agent or know what to expand.

Frequently asked questions

How long does a build like this take?

For a single well-scoped use case, a small team can reach suggestion mode in a few weeks and autonomous on the easy intents shortly after. Most of the early time goes to scoping and the eval set, not to the agent itself — which is exactly where it should go.

Why ship in suggestion mode instead of going live?

Suggestion mode keeps blast radius at zero while the agent earns trust, and every human correction becomes a new eval case. It is the cheapest way to gather real-world ground truth before you let an agent act on its own.

What makes a use case a good first agent?

High volume, narrow scope, well-bounded actions, and mostly reversible outcomes. Repetitive tier-1 tasks with clear backend lookups are ideal; ambiguous, high-stakes, or irreversible work should wait.

How do I know when to expand to more intents?

Let the metrics decide. When an intent sustains high deflection and accuracy with stable customer satisfaction, add the next one using the same scope-eval-tools-rollout loop. Expansion is repetition of a proven pattern, not a new project each time.

From support backlog to answered phone lines

CallSphere runs this exact problem-to-outcome pattern on voice and chat — scoped agents that answer every call, use tools mid-conversation, and book real work, with the metrics to prove it. See a live walkthrough at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.