Before an AI Agent Answers a Support Ticket Unsupervised, Build the Test Set Out of Last Year's Zendesk Reopens

It is 4:52 on a Thursday afternoon and your head of support forwards you a reopened ticket. Two days earlier an assistant drafted the reply and a Tier 1 rep sent it after one glance: yes, single sign-on enforcement can be turned on for individual users while the rest of the workspace keeps password login. It cannot. Your product enforces SSO at the workspace level or not at all. The customer's IT director had already repeated that answer to their security review board, the board had already written it into a rollout plan, and the reply now sitting in your inbox has the customer's CTO copied in — eleven weeks before an $84,000 renewal.

Nothing in that sequence looked like a dramatic AI failure. The answer was fluent, correctly formatted, cited a help-center article, and was wrong on one word. What failed was the decision to let it go out before anyone had ever measured how often it gets that exact kind of question wrong.

The thing that broke was the rollout decision, not the answer

Most B2B software companies between $3M and $15M in recurring revenue tried an AI answer assistant somewhere between 2024 and 2025. The rollout usually went the same way: someone connected it to the Zendesk or Intercom help center, watched twenty replies over a coffee, said "that's pretty good", and turned it on for the whole Tier 1 queue. There was no scoring, no record of what the agent looked at before it answered, and no defined moment at which it earned more authority. The gut-feel pilot is the workaround everybody pretends is fine.

The problem with gut feel in support is that the failures are rare and expensive rather than common and cheap. Ninety-four out of a hundred tickets are password resets, invoice copies, "where do I export this", and seat changes. The other six are the ones where a wrong answer becomes a renewal conversation with a CTO on it, a security questionnaire that has to be corrected in writing, or a customer who builds a workflow on a capability you do not have.

An evaluation set is simply a folder of your own past tickets where you already know what the right answer was, used to score an AI agent on real cases before a single customer sees its work. That is the whole idea, and it is closer to the way you would test a new support hire: hand them last quarter's hard tickets and read what they write.

What actually changed by mid-2026

Through 2025 you could run an AI agent, but you mostly could not see what it did. You got the final reply and nothing else. During the first half of 2026 the tooling around watching and scoring agents matured to the point where it now gates the rollout rather than decorating it. Two things are new and both matter to an owner.

First, you can replay a decision step by step: which help-center article the agent opened, which account record it looked up, whether it checked the customer's plan tier before quoting a feature, whether it tried to issue a credit. Second, you can score the agent against a fixed set of your own historical cases and get a number that moves when you change something — so "we made it better" becomes "wrong answers on entitlement questions fell from 31 to 6 out of 120".

Alongside that, the July 2026 enterprise governance work from Anthropic added spend limits, alerts at 75% and 90% of budget, and per-team model defaults. That sounds like finance plumbing, and it is, but it is also what lets you run a scoring pass over a few hundred tickets without an unbounded bill.

flowchart TD
  A["Pull 400 closed tickets from Zendesk"] --> B["Agent answers each one with the customer reply hidden"]
  B --> C{"Does it match what the human actually sent?"}
  C -->|No| D["Replay the steps: which article, which account lookup, which plan check"]
  D --> E["Fix the source article or tighten the rule"]
  E --> B
  C -->|Yes| F["Draft-only in the live queue for two weeks"]
  F --> G{"Support lead approves 95% of drafts unedited?"}
  G -->|Yes| H["Auto-send password, invoice-copy and seat-change tickets only"]
  G -->|No| D

Your test set already exists — it is the reopens and the CSAT ones

You do not have to write test cases. You have to go find them. Four places in a normal support history hold everything you need, and a Zendesk or Intercom export gets you all four in an afternoon.

• Reopened tickets. Any ticket closed and then reopened by the customer within seven days is a case where the first answer was insufficient or wrong. These are gold. Pull twelve months of them.
• CSAT 1s and 2s with a comment. The comment usually says exactly what the answer got wrong.
• Anything escalated from Tier 1 to a support engineer. These define the boundary you actually care about: the agent's job on these is to escalate, not to answer.
• Your macros that nobody trusts. Every support team has three or four canned replies that are quietly out of date since a release. Those are the answers the agent will confidently repeat.

Aim for roughly 400 cases: about 250 routine ones that should be answered, 100 that should have been escalated, and 50 that are wrong-answer landmines — entitlement questions, data-retention questions, anything about SSO or SCIM, anything about your uptime commitment, anything that touches a signed order form. Weight the scoring so a confidently wrong answer on the landmine 50 counts far worse than a slightly clumsy answer on the routine 250.

Three grades, not two

Pass and fail is not enough. Score every case into three buckets, because the third is the one that protects you.

Correct. The substance matches what your human sent, allowing for different wording. Wrong. It contradicts the human answer, or invents a setting, plan limit, or commitment. Should have escalated but answered anyway. This is the bucket that predicts renewal damage. An agent that says "let me bring in a support engineer on this one" on a hard case is doing its job perfectly. An agent that never says it is the one that copies a CTO into your Thursday.

Run the whole 400 before you turn anything on, then again after every help-center change and every model change. Twenty minutes reviewing the failures beats a month of anecdotes.

What the wrong-answer rate is actually worth

Here is the arithmetic, with assumptions stated so you can substitute your own numbers. Suppose your support desk handles 6,000 tickets a quarter, 58% of which are Tier 1 eligible, and a wrong answer costs you 22 minutes of a support engineer plus 15 minutes of a customer success manager writing the apology — call it $95 fully loaded. Suppose the ungated agent scores 11% wrong on your 400-case set and the gated one, after two rounds of fixes, scores 3%.

That table deliberately ignores the expensive part, because you cannot honestly put a number on it: one wrong entitlement answer inside a security review at an enterprise account can cost more than a year of that rework line. The point of the test set is not the $26,505. It is that you now have a number at all, measured on your own tickets, that you can show your board and re-measure next quarter.

Where a human stays, permanently

Do not automate anything that produces a written commitment. Refunds and credits, uptime-credit claims, security questionnaires such as SIG Lite or CAIQ, anything that quotes a data-retention window, anything that touches a signed order form or master service agreement, and anything from an account in an active renewal window. Those go to a person even when the agent is right, because the customer will forward the reply and it will be read as a company statement.

Keep a human on the escalation judgement for at least a quarter. Have your support lead spot-check twenty auto-sent replies a week and specifically ask: should this one have gone to an engineer? That is a thirty-minute Friday habit, and it catches drift that no dashboard will show you.

And be honest about the ceiling. An agent scored on last year's tickets knows last year's product. Every release that changes a limit, a plan boundary, or a setting name invalidates part of your test set. Put "update the eval cases" on the release checklist next to "update the help center", or the whole exercise decays in two quarters.

Frequently asked questions

How long does it actually take to build the 400-case set?

Two working days for a support lead who knows the queue. One day exporting and sorting reopens, CSAT 1s and 2s, and escalations; one day picking the 50 landmine cases and writing down what the correct answer is. The cases already exist — you are curating, not authoring.

Can I skip this and just start in draft-only mode?

Draft-only is safer than auto-send, but it does not tell you what you need to know. Reps under queue pressure approve drafts quickly, so a plausible wrong answer still goes out. Draft-only is the second gate, not the first.

What score is good enough to turn on auto-send?

There is no industry number, and be suspicious of anyone who gives you one. Set it against your humans: score twenty of your own reps' historical replies the same way, and require the agent to beat that baseline on routine cases while never answering a landmine case it should have escalated.

Does this apply to the AI feature inside my product too?

Yes, and more so. A support reply is seen by one customer; a wrong answer inside your product is seen by all of them, and it is your product making the claim. The same test set discipline applies, built from your own support and solutions-engineering history.

What to do on Monday

Open Zendesk, filter for tickets reopened by the customer within seven days over the last twelve months, and export them. That single list is the spine of your test set; everything else follows from having it.

The same discipline applies at the front door, where a prospect first calls or opens the chat widget. CallSphere builds AI voice and chat agents that answer business lines and website chat, book demos, and capture leads around the clock — and the sensible way to bring one in is the way described above: score it against your own past calls and chats first, then widen its authority only once the numbers say so.