Don’t Let an Agent Take an Obituary Until It Passes 120 of Your Own Calls — Including the Ones Where the Name Was Misspelled

What happens the first time your AI agent spells a dead woman's name wrong?

Not hypothetically. Concretely: it takes the obituary by phone from a funeral director at 4:40 on a Friday, hears "Kathryn," sets "Katherine," and 3,900 copies go out Saturday morning with the wrong name on a woman whose family has taken this paper since 1961. You run a correction. You refund the obit. You lose the funeral home, which sends you 60 to 90 paid obituaries a year at $180 to $400 apiece, and you do not get it back, because the funeral director has to face that family at the visitation and you do not.

That single scenario is why most publishers who tried an answering agent in 2024 turned it off inside a month. It is also the exact problem the 2026 tooling was built for. Agent evaluation is the practice of scoring an AI agent against a set of your own real past cases, with a known correct answer for each one, before you let it speak to a customer — and then watching, case by case, what it actually did rather than whether it sounded confident.

Pull 120 calls before you write a single instruction

Every paper has this material already. Your phone system has recordings or at minimum a call log; your obituary folder has the finished, published, family-approved text of every obit you ran; your billing system has what each one cost. Those three things, joined together, are a test set.

Pull 120 calls from the past year, weighted the way your week actually looks. For a weekly with a Tuesday noon obit deadline, that is roughly: 45 obituary intakes from funeral homes, 25 missed-delivery complaints, 15 vacation stops and restarts, 12 classified line ads, 10 calendar and community-submission calls, 8 legal notice deadline questions, and 5 calls that were something else entirely — a reader who wanted 1987 microfilm, somebody looking for the sports editor, a wrong number that turned into a news tip.

Then build the answer key. For an obituary intake, it is eleven fields: legal name and exact spelling, age, date of death, town of residence, funeral home and its billing account, visitation date, time and location, service date and time, preceded-in-death and survived-by names, photo yes or no, and the word count with the price it produces. That is not a technical exercise. That is your obituary clerk with a legal pad and two afternoons.

Run it in the shadows first

Shadow mode means the agent takes the call in parallel and produces its answer, but a human still sends the result. Nobody outside the building knows it is running. For a week or two, every call your front counter takes also gets handled invisibly, and at the end of each day you have two versions to compare: what your clerk wrote down and what the agent wrote down.

This is where the 2026 tooling earns its keep. It is not just a pass-or-fail score. You can see the whole sequence — which field it asked about first, where it accepted an answer it should have questioned, the exact moment it took "Kathryn with a K-A-T-H" and wrote "Katherine." That step-by-step view is what turns a vague sense that "it's pretty good" into a specific fix. The Claude Enterprise governance update on 2 July 2026 added the other half of what a small business needs here: a cost and usage dashboard with spend limits and alerts at 75 and 90 percent, so a runaway agent shows up as an alert rather than as a surprise bill.

flowchart TD
  A["Pull 120 recorded intake calls"] --> B["Build answer key: 11 fields per obituary"]
  B --> C["Agent runs in shadow, clerk still sends"]
  C --> D{"All 11 fields correct on 118 of 120?"}
  D -->|No| E["Fix the misses, add failed calls to the set"]
  E --> C
  D -->|Yes| F["Live for after-hours obituary intake only"]
  F --> G["Widen to daytime calls, then to price quotes"]
  G --> D

Widen authority one notch at a time

The mistake is treating the go-live as a switch. It is a ladder, and each rung has its own bar to clear.

Rung one: after-hours message taking only. The agent answers at 6:15 p.m., collects the caller's name, number and reason, and emails the front counter. Nothing it says commits you to anything. Rung two: missed delivery and vacation stops — low stakes, high volume, and your district manager gets a clean list at 7 a.m. instead of nine voicemails. Rung three: obituary intake with a mandatory proof, meaning the agent takes the details and immediately emails the funeral home a proof to approve before anything is set. Rung four: quoting the price and confirming the deadline. Rung five, which many papers should never climb, is committing to a run date without a human touching it.

Each rung needs the test set rerun, because widening authority changes what a mistake costs. An agent that is 96 percent accurate is fine at rung one and a liability at rung four.

The arithmetic: what a two-point error rate is worth

Assume 26 paid obituaries a week at an average of $265, which is about $344,000 a year and, at most weeklies, one of the top three revenue lines. Assume your current human error rate on intake — a wrong middle initial, a visitation time off by an hour, a missing survivor — is 2 percent, because you have been counting and it is.

Notice what this table is not claiming. It does not claim the agent is flawless; it claims a tested agent with a forced proof step beats an interrupted human at 4:40 on a Friday. And notice the line that is missing, because it cannot be honestly estimated: the funeral home relationship. One lost account at 70 obituaries a year is $18,550, which dwarfs everything in the table. That is the number the test set is really protecting.

Where you keep a human, permanently

Family-submitted obituaries at the front counter stay human. A widow standing at your window with a handwritten page is not a workflow. The death of a public figure — a sitting mayor, a longtime coach, a former publisher — goes straight to the editor, because that is a news story with an obituary attached, not an obituary. Anything where the caller is crying goes to a person, and the agent should be built to hand off the second it hears it.

On the business side, keep a human on deadline commitments. Only your paginator and your press foreman know whether an obit that lands at 11:50 on Tuesday can still make the book. And keep a human on credit — an agent that agrees to run $400 of obituary for a funeral home 90 days past due has cost you more than it saved.

Start Monday with 20 calls, not 120

Full test sets are how this stalls. Pull twenty obituary calls, write the answer key for those twenty, and run any of the current agents against them cold, with no tuning at all. You will get a number within an afternoon. If it gets 14 of 20 completely right, you know the shape of the work ahead. If it gets 19, you have a much shorter road than you thought. Either way you now have a measurement instead of an opinion, which is the entire difference between 2026 and the version of this you tried two years ago.

Frequently asked questions

How long does building a test set actually take?

For twenty calls with an eleven-field answer key, an afternoon. For the full 120, budget two to three days of your obituary clerk's time spread over a couple of weeks. It is the least glamorous work in this entire subject and it is the only part that determines whether the rollout succeeds.

Do I need permission to use our recorded calls this way?

Check your state's recording law and whatever your phone greeting already says — if you are recording for quality purposes you likely have what you need, but a dozen states require all-party consent and your greeting has to reflect it. Separately, California SB 53 and Texas TRAIGA both took effect 1 January 2026, and several other states have their own AI statutes; federal preemption is unsettled as of July 2026, so your state's rules still bind. Ask your attorney once, in writing, and keep the answer.

What accuracy is good enough to go live?

It depends entirely on the rung. For after-hours message taking, if it captures name, callback number and reason on 19 of 20, go. For anything that quotes a price or commits to a run date, you want essentially no misses on your test set and a mandatory proof step behind it, because the errors that survive to print are the expensive ones.

What if the agent is better than my clerk on the test but worse on live calls?

That gap is real and it usually means your test set is too clean — you pulled the calls where the funeral director was organized. Deliberately include your worst recordings: the cell phone in a parking lot, the caller with a heavy accent, the one where three people talk at once. Those are the calls that decide it.

One note on the phone

Everything above assumes you have something answering the line in the first place, and that it can be measured. CallSphere builds AI voice and chat agents that answer a business phone line and website chat, take details, book appointments and capture leads around the clock — which for a paper means the 6:15 p.m. obituary call and the 7 a.m. missed-delivery call both get picked up. Run it the way this piece describes: shadow first, your own past calls as the test, and authority widened one rung at a time.