Before Your Parts Desk Agent Quotes a Seal Kit for Serial 4471, Score It Against 400 Closed Orders

The credit memo was on the controller's desk before the coffee finished brewing, and by then everybody in the building knew the story. A maintenance planner at a cheese plant outside Fresno had called the parts line about a machine your company built in 2013. Inside sales pulled the serial up in Epicor, found the drive-end seal kit on the as-shipped bill of materials, and put it on an overnight for $214 in freight. Wrong kit. That serial picked up a field change in 2017 that moved it to a larger shaft, and the seal went with it. The machine stayed down another day and a half, the customer got a goodwill credit, and your service manager spent forty minutes on the phone being polite.

Nobody was careless. The as-built history for that machine lives in three places: the original bill of materials in the ERP, the change order folder in SolidWorks PDM, and a field service report in a shared drive nobody has opened since the tech who wrote it left for a competitor. Every industrial equipment builder I have walked through has some version of this. It is also the exact job owners are now being told to hand to an agent — and this particular mistake is invisible until the return shows up.

The parts desk between 7:00 and 9:30 in the morning

Your aftermarket coordinator opens with roughly forty open lines and a voicemail box. The inquiries arrive three ways: the parts line, the shared parts inbox, and a photo texted to a regional service manager who forwards it with "customer needs this today." The typical message is not a part number. It is "we're leaking at the drive end, serial 4471, what do we need." Sometimes there is a nameplate photo, sometimes there is a nameplate photo of a different machine on the same line.

To answer it properly the coordinator opens the as-shipped bill of materials, checks the change order log for that serial, cross-references the gearmotor tag because the seal may belong to the gearbox vendor and not to you, checks the superseded-part table because the number on the 2013 drawing was replaced twice, and then calls the field tech who was last on that site. Fifteen to twenty-five minutes when it goes well. When it does not, it becomes a credit memo.

The workaround everyone pretends is fine: two people who have been there fifteen years know most of these answers cold. One of them is 61. Aftermarket parts carry the gross margin machine sales cannot, and the desk producing it is staffed like a cost center and documented like a hobby.

What changed in 2026: you can watch every step and grade it before it talks to a customer

The 2024 version of this idea was a chat box bolted to your website that gave a fluent answer and no way to know whether it was right until the return authorization came back ninety days later. The 2025 version could search your documents, which helped, and still gave you no scoreboard.

What matured in 2026 is the boring half: measuring an agent against real cases and watching what it actually did, step by step, before you widen what it is allowed to touch. An agent test set is nothing more exotic than a folder of your own closed orders where you already know the correct answer, used to score a new system before it is ever allowed to speak to a customer. You run it unattended against those cases, you read the trail of what it opened and what it decided, and only then does it get to draft a reply — and later, maybe, send one.

flowchart TD
  A["Pull 400 closed parts orders from the ERP"] --> B["Mark the part that stayed shipped, no return in 90 days"]
  B --> C["Run the agent on each inquiry, unattended"]
  C --> D{"Exact part number and correct as-built revision?"}
  D -->|"Below the bar"| E["Read the step trail, fix the source record"]
  E --> C
  D -->|"At or above the bar"| F["Agent drafts, coordinator sends"]
  F --> G["Re-score every Friday on that week's orders"]
  G --> C

Building the test set out of orders you already shipped

Take the last twelve to eighteen months of parts orders and pull three to five hundred lines where a human made a judgment and it held. "Correct" is not what somebody typed in the quote — it is the part that stayed shipped, with no return and no credit inside ninety days. That one rule turns your order history into an answer key without anyone writing an answer key.

Then load it with the ugly cases, because the easy ones prove nothing. Serials with field change orders. Bought-out components where the right answer is the gearmotor vendor's number and not yours. The two models you discontinued in 2016. Machines a customer's own maintenance team modified without telling you, which the field service reports will reveal. And twenty to thirty cases where the correct action was not an answer — it was "send me a nameplate photo" or "this needs the service manager."

Pull cases from the weeks that actually hurt: the July shutdown stretch and the two weeks before Christmas, when customers do their teardowns and your parts volume spikes while your desk is at half staff. An agent that scores well in April and falls apart on shutdown-week questions has not been tested.

The rubric is five columns, and one of them is "I don't know"

Grade every case on five things, and grade them separately so you can see where it breaks:

• Part number — exact, not close.
• As-built revision — did it apply the change orders that hit this serial, or did it read the original bill of materials and stop.
• Quantity and unit of measure — a kit versus four individual seals is a real failure that looks like a rounding error.
• Citation — which record or drawing it pulled from, so a human can check in ten seconds.
• Correct refusal — did it stop and escalate when the as-built was ambiguous. Score a good refusal as a win.

Then read the step trail on a sample. You are looking for right answers reached for the wrong reason. An agent that ignored the change order log and happened to be correct on an unmodified machine is not accurate. It is lucky, and it will be unlucky on serial 4471.

What a four percent wrong-part rate actually costs

Assumptions, stated plainly and illustrative: nine hundred parts orders a year through the desk, a current wrong-part rate of four percent, and a loaded internal labor cost of $70 an hour. One in six wrong shipments escalates into a downtime credit or a goodwill spare.

Roughly twenty thousand dollars a year on a desk that size, and it excludes the orders you never saw because the planner started calling a parts broker instead. Running an assistant across nine hundred text inquiries a year is cheap enough that freight dominates this table; frontier models are down roughly tenfold from 2025. The real build cost is your own people spending a few weeks cleaning the superseded-part table and getting the change order log into one place — work you needed anyway.

Where the agent should never get authority

Do not let it commit a ship date on a long-lead item. Gearmotors, servo drives and variable frequency drives move on their own schedule, and a date promised by a machine is still a promise from you.

Do not let it answer on anything inside the safety circuit — light curtains, safety relays, interlock switches. Swapping a device changes the rating of that circuit and puts your risk assessment and your conformity file into play. That question goes to a named engineer, every time, no exceptions, and the agent's only correct move is to route it.

Do not let it decide warranty. Whether a bearing that failed at fourteen months is a warranty claim or an alignment problem at the customer's plant is a commercial judgment about a relationship, and your service manager gets paid to make it.

And do not point it at machines older than your records. If your as-built data starts in 2009 and the machine is from 1997, the honest answer is a photo and a phone call. The fifteen-year veteran does not get replaced by this. He gets interviewed by it — one hour a week on the cases the agent got wrong is the cheapest documentation project you will ever run.

Frequently asked questions

How many cases do I need before the score means anything?

Two to three hundred real closed orders beats a thousand invented ones. The mix matters more than the count: if every case is a current-model machine with clean records, the score is flattering and useless. Make a quarter of the set field-changed serials, discontinued models and bought-out components.

Our as-built records are a mess. Do we have to clean everything first?

No, and trying to will kill the project. Run the scoring on the mess. The failures cluster, and they hand you the thirty or forty serial numbers whose records are actually costing you money. Clean those, re-score, move on.

Who owns this internally? Do I need to hire someone?

Your aftermarket or parts manager owns it, and it is spreadsheet work, not engineering work. The one thing you should not do is hand it to applications engineering, who are already the bottleneck on quotes and will do it last.

What score is high enough to let it reply to a customer directly?

It depends on how reversible the mistake is. Drafting a reply the coordinator sends is low risk at a modest score. Sending a part number that triggers an overnight shipment is a different bar, and most builders keep a human on the send button for two quarters, then remove it only for current-model, unmodified machines.

The same discipline applies to the parts line itself. When the coordinator is already on a call, the next planner with a down machine hears a voicemail greeting. CallSphere builds voice and chat agents that answer that line around the clock, take the serial number, the symptom and the site, and book the callback so the inquiry lands in your system instead of someone's memory. Score it the way you would score anything else that talks to your customers.