By Sagar Shankaran, Founder of CallSphere
Build the 400-case test set out of your own reopened Zendesk tickets and CSAT 1s and 2s before letting an AI agent answer B2B SaaS support tickets alone.
Key takeaways
It is 4:52 on a Thursday afternoon and your head of support forwards you a reopened ticket. Two days earlier an assistant drafted the reply and a Tier 1 rep sent it after one glance: yes, single sign-on enforcement can be turned on for individual users while the rest of the workspace keeps password login. It cannot. Your product enforces SSO at the workspace level or not at all. The customer's IT director had already repeated that answer to their security review board, the board had already written it into a rollout plan, and the reply now sitting in your inbox has the customer's CTO copied in — eleven weeks before an $84,000 renewal.
Nothing in that sequence looked like a dramatic AI failure. The answer was fluent, correctly formatted, cited a help-center article, and was wrong on one word. What failed was the decision to let it go out before anyone had ever measured how often it gets that exact kind of question wrong.
Most B2B software companies between $3M and $15M in recurring revenue tried an AI answer assistant somewhere between 2024 and 2025. The rollout usually went the same way: someone connected it to the Zendesk or Intercom help center, watched twenty replies over a coffee, said "that's pretty good", and turned it on for the whole Tier 1 queue. There was no scoring, no record of what the agent looked at before it answered, and no defined moment at which it earned more authority. The gut-feel pilot is the workaround everybody pretends is fine.
The problem with gut feel in support is that the failures are rare and expensive rather than common and cheap. Ninety-four out of a hundred tickets are password resets, invoice copies, "where do I export this", and seat changes. The other six are the ones where a wrong answer becomes a renewal conversation with a CTO on it, a security questionnaire that has to be corrected in writing, or a customer who builds a workflow on a capability you do not have.
An evaluation set is simply a folder of your own past tickets where you already know what the right answer was, used to score an AI agent on real cases before a single customer sees its work. That is the whole idea, and it is closer to the way you would test a new support hire: hand them last quarter's hard tickets and read what they write.
Through 2025 you could run an AI agent, but you mostly could not see what it did. You got the final reply and nothing else. During the first half of 2026 the tooling around watching and scoring agents matured to the point where it now gates the rollout rather than decorating it. Two things are new and both matter to an owner.
First, you can replay a decision step by step: which help-center article the agent opened, which account record it looked up, whether it checked the customer's plan tier before quoting a feature, whether it tried to issue a credit. Second, you can score the agent against a fixed set of your own historical cases and get a number that moves when you change something — so "we made it better" becomes "wrong answers on entitlement questions fell from 31 to 6 out of 120".
Hear it before you finish reading
Talk to a live CallSphere AI voice agent for IT support in your browser — 60 seconds, no signup.
Alongside that, the July 2026 enterprise governance work from Anthropic added spend limits, alerts at 75% and 90% of budget, and per-team model defaults. That sounds like finance plumbing, and it is, but it is also what lets you run a scoring pass over a few hundred tickets without an unbounded bill.
flowchart TD
A["Pull 400 closed tickets from Zendesk"] --> B["Agent answers each one with the customer reply hidden"]
B --> C{"Does it match what the human actually sent?"}
C -->|No| D["Replay the steps: which article, which account lookup, which plan check"]
D --> E["Fix the source article or tighten the rule"]
E --> B
C -->|Yes| F["Draft-only in the live queue for two weeks"]
F --> G{"Support lead approves 95% of drafts unedited?"}
G -->|Yes| H["Auto-send password, invoice-copy and seat-change tickets only"]
G -->|No| D
You do not have to write test cases. You have to go find them. Four places in a normal support history hold everything you need, and a Zendesk or Intercom export gets you all four in an afternoon.
Aim for roughly 400 cases: about 250 routine ones that should be answered, 100 that should have been escalated, and 50 that are wrong-answer landmines — entitlement questions, data-retention questions, anything about SSO or SCIM, anything about your uptime commitment, anything that touches a signed order form. Weight the scoring so a confidently wrong answer on the landmine 50 counts far worse than a slightly clumsy answer on the routine 250.
Pass and fail is not enough. Score every case into three buckets, because the third is the one that protects you.
Correct. The substance matches what your human sent, allowing for different wording. Wrong. It contradicts the human answer, or invents a setting, plan limit, or commitment. Should have escalated but answered anyway. This is the bucket that predicts renewal damage. An agent that says "let me bring in a support engineer on this one" on a hard case is doing its job perfectly. An agent that never says it is the one that copies a CTO into your Thursday.
Run the whole 400 before you turn anything on, then again after every help-center change and every model change. Twenty minutes reviewing the failures beats a month of anecdotes.
Here is the arithmetic, with assumptions stated so you can substitute your own numbers. Suppose your support desk handles 6,000 tickets a quarter, 58% of which are Tier 1 eligible, and a wrong answer costs you 22 minutes of a support engineer plus 15 minutes of a customer success manager writing the apology — call it $95 fully loaded. Suppose the ungated agent scores 11% wrong on your 400-case set and the gated one, after two rounds of fixes, scores 3%.
| Line | Ungated | Gated |
|---|---|---|
| Tier 1 tickets per quarter | 3,480 | 3,480 |
| Wrong-answer rate on the test set | 11% | 3% |
| Wrong answers reaching customers | 383 | 104 |
| Rework at $95 each | $36,385 | $9,880 |
| Quarterly difference | $26,505 | |
That table deliberately ignores the expensive part, because you cannot honestly put a number on it: one wrong entitlement answer inside a security review at an enterprise account can cost more than a year of that rework line. The point of the test set is not the $26,505. It is that you now have a number at all, measured on your own tickets, that you can show your board and re-measure next quarter.
Still reading? Stop comparing — try CallSphere live.
See the IT support AI agent handle a real call — complete, industry-specific, and live in your browser. No signup.
Do not automate anything that produces a written commitment. Refunds and credits, uptime-credit claims, security questionnaires such as SIG Lite or CAIQ, anything that quotes a data-retention window, anything that touches a signed order form or master service agreement, and anything from an account in an active renewal window. Those go to a person even when the agent is right, because the customer will forward the reply and it will be read as a company statement.
Keep a human on the escalation judgement for at least a quarter. Have your support lead spot-check twenty auto-sent replies a week and specifically ask: should this one have gone to an engineer? That is a thirty-minute Friday habit, and it catches drift that no dashboard will show you.
And be honest about the ceiling. An agent scored on last year's tickets knows last year's product. Every release that changes a limit, a plan boundary, or a setting name invalidates part of your test set. Put "update the eval cases" on the release checklist next to "update the help center", or the whole exercise decays in two quarters.
Two working days for a support lead who knows the queue. One day exporting and sorting reopens, CSAT 1s and 2s, and escalations; one day picking the 50 landmine cases and writing down what the correct answer is. The cases already exist — you are curating, not authoring.
Draft-only is safer than auto-send, but it does not tell you what you need to know. Reps under queue pressure approve drafts quickly, so a plausible wrong answer still goes out. Draft-only is the second gate, not the first.
There is no industry number, and be suspicious of anyone who gives you one. Set it against your humans: score twenty of your own reps' historical replies the same way, and require the agent to beat that baseline on routine cases while never answering a landmine case it should have escalated.
Yes, and more so. A support reply is seen by one customer; a wrong answer inside your product is seen by all of them, and it is your product making the claim. The same test set discipline applies, built from your own support and solutions-engineering history.
Open Zendesk, filter for tickets reopened by the customer within seven days over the last twelve months, and export them. That single list is the spine of your test set; everything else follows from having it.
The same discipline applies at the front door, where a prospect first calls or opens the chat widget. CallSphere builds AI voice and chat agents that answer business lines and website chat, book demos, and capture leads around the clock — and the sensible way to bring one in is the way described above: score it against your own past calls and chats first, then widen its authority only once the numbers say so.
Written by
Sagar Shankaran· Founder, CallSphere
LinkedInSagar Shankaran is the founder of CallSphere, where he builds production AI voice and chat agents deployed across healthcare, hospitality, real estate, and home services. He writes about agentic AI, LLM engineering, and shipping voice agents that handle real calls in production.
See how AI voice agents work for your industry. Live demo available -- no signup required.
The 200-call test set a treatment program should build from its own recordings, the five things to score, and how to widen an agent's authority safely.
Model routing sends routine product requests to a cheap model and hard ones to a strong one. Here is who draws the line in a B2B software company, and how.
How an overnight agent reconciles Stripe to NetSuite, drafts the dunning emails and flags non-renewal notice windows before your revenue analyst logs in at 7am.
How 2026 live speech translation cuts truck rolls at a 4,000-subscriber fiber carrier serving Marshallese, Spanish, Hmong and Haitian Creole households.
Build an AI test set from closed mortgage files: the 1008 qualifying income is the answer key, QC defects are the hard cases, and the breakdown beats the score.
How a caterer builds a 240-case test set from real BEOs and inquiry email, what it costs to grade it, and the category floors to hit before going live.
© 2026 CallSphere Inc. All rights reserved.
Made within San Francisco
Watch how CallSphere handles real customer calls, schedules appointments, and processes payments — live.
Try Live DemoBook a DemoCalculate Your ROI