End-to-End: Shipping a Refined Claude Skill in a Week

Abstract advice about evals and descriptions only lands when you watch it on a real task. So this post follows one skill from a Monday-morning complaint to a Friday-afternoon release. The problem: a support team kept asking Claude to "summarize this ticket and draft a reply," and the results swung wildly — sometimes a tight summary with a draft, sometimes a wall of text, sometimes no draft at all. The team had a prompt but no skill, no tests, and no idea why it varied. We'll turn that into a refined, measured Agent Skill using skill-creator, and you'll see every decision, including the ones that didn't work the first time.

Key takeaways

• A realistic skill build is iterative: draft, eval, read transcripts, fix the right layer, repeat.
• Most early failures are description and scoping problems, not model capability problems.
• Writing the eval set before the skill body forces you to define "good" before you can fool yourself.
• Variance — not the best run — is the metric that tells you whether the skill is ready to ship.
• Shipping means a versioned skill, a recorded baseline, and a rollback plan, all achievable in days.

Day 1: turning a complaint into a spec

The first move is not writing instructions — it is writing examples of success and failure. We collected ten real tickets and, for each, wrote what an ideal output looks like: a three-sentence summary, the customer's core ask, and a draft reply under 120 words in the team's tone. We also wrote what failure looks like: a summary longer than the ticket, a reply that invents a refund policy, or no draft at all. This labeled set becomes the eval. An Agent Skill is a folder of instructions and resources Claude loads when a task matches its description, so before scaffolding the folder we knew exactly what behavior we were optimizing toward.

Day 2: scaffold, then watch it fail honestly

We used skill-creator to scaffold ticket-triage and wrote a first SKILL.md. Then we ran the eval — five runs per ticket — and read the transcripts. The flow we followed, including the loop back when results disappointed, looked like this.

flowchart TD
  A["10 real tickets + ideal outputs"] --> B["Write eval set first"]
  B --> C["skill-creator scaffolds ticket-triage"]
  C --> D["Run eval: 5 runs per ticket"]
  D --> E{"Meets bar across runs?"}
  E -->|No| F["Read transcripts, find failing layer"]
  F --> G["Fix description, scope, or resource"]
  G --> D
  E -->|Yes| H["Version, record baseline, ship"]

The first run scored poorly — drafts appeared only about half the time. The transcripts made the cause obvious: the description said "summarize and optionally draft a reply," and Claude reasonably treated the draft as optional. We removed "optionally." Recall on the draft jumped immediately. This is the pattern of real skill work: the model was not failing; our instructions were ambiguous, and the eval made the ambiguity visible.

Day 3: the resource fix that mattered most

The remaining failure was tone and invented policies. The draft replies sometimes promised refunds the company doesn't offer. The fix was not a better prompt — it was a resource. We added a short policy file the skill loads and instructs Claude to quote from rather than improvise. Here is the relevant slice of the refined skill.

---
name: ticket-triage
description: >
  Summarize a support ticket in 3 sentences and ALWAYS draft a reply
  under 120 words in the brand voice. Trigger when the user shares a
  ticket and asks to summarize, triage, or draft a response.
---

## Steps
1. Summarize the ticket in exactly 3 sentences.
2. State the customer's single core ask in one line.
3. Draft a reply (<120 words). Use ONLY claims found in
   resources/policy.md. If the answer isn't in policy.md, say you'll
   escalate rather than inventing a policy.

## Resources
- resources/policy.md  (refunds, SLAs, escalation rules)

Grounding the draft in policy.md with an explicit "don't invent" instruction eliminated the fabricated-refund failures. After this change, the eval passed the bar on nine of ten tickets across all five runs.

Day 4: measuring readiness, not luck

One ticket still failed intermittently — passing on three runs, failing on two. The tempting move was to ship and call it a fringe case. Instead we treated variance as the gate. We ran twenty additional runs on that ticket and found the failure correlated with very long ticket threads where the summary blew past three sentences. We tightened the instruction to truncate context, and the variance collapsed. The takeaway: a skill that passes on average but swings wildly is not ready; consistency across runs is the real signal.

Common pitfalls in an end-to-end build

• Writing the skill before the eval. You'll define success to match whatever the skill already does. Write the labeled examples first.
• Fixing the wrong layer. When a draft is missing, people rewrite the whole prompt. Read the transcript — often one word like "optionally" is the bug.
• Improvising facts the skill should ground. If outputs invent policy, add a resource file and forbid improvisation, rather than hoping a sterner tone helps.
• Shipping on the best run. Demos lie. Gate on the worst-case and variance across many runs.
• No baseline recorded. Without the numbers from launch day, you can't tell whether next month's edit helped or hurt.

Ship a refined skill in five steps

1. Collect 8–12 real inputs and write ideal and failing outputs for each.
2. Encode those as a skill-creator eval before scaffolding the skill.
3. Draft SKILL.md, run the eval with multiple runs per case, and read transcripts to locate the failing layer.
4. Fix description, scope, or resources — adding grounding files where outputs need facts.
5. Gate on variance, version the skill, record the baseline, and keep the prior version for rollback.

Before and after the refinement

Aspect	Day 1 (prompt only)	Day 5 (refined skill)
Draft included	~50% of the time	Every run
Invented policy	Occasional	Eliminated via grounding
Output length	Unbounded	3-sentence summary, <120-word reply
Confidence to ship	None (no measurement)	Recorded baseline + rollback

Frequently asked questions

How long does a realistic skill take to ship?

For a contained workflow like this, a few focused days. The eval set is the time investment; once it exists, iteration is fast.

What was the most impactful single change?

Adding a grounded resource file and forbidding improvisation. It converted a confident-but-wrong skill into a trustworthy one.

Do I always need real production inputs?

Strongly preferred. Synthetic inputs miss the messy long threads and edge phrasings that cause most real failures.

When is a skill actually done?

When it passes your bar consistently across many runs, has a recorded baseline, and a documented rollback — not when it nails a single demo.

From skills to live conversations

CallSphere runs this same problem-to-shipped loop for voice and chat agents — grounded in your policies, measured across runs, and refined until they answer every call and book work reliably. See the result at callsphere.ai.

---

Source & attribution: This is an independent, original explainer inspired by Anthropic's coverage on the Claude blog. Claude, Claude Code, Claude Cowork, Claude Opus, and the Model Context Protocol are products and trademarks of Anthropic. CallSphere is not affiliated with or endorsed by Anthropic.