Before an Agent Touches a Title Commitment, Feed It 400 Closed Files and Watch Where It Misses the Judgment Lien

Run a name search on a seller called John A. Miller in a county of any size and you get back somewhere between forty and two hundred hits. Judgments, state tax liens, child support, an old federal tax lien, three UCC filings, a divorce decree, two probate matters. Your title examiner clears almost all of them in an afternoon, because almost none of them are your John A. Miller. The share of name hits that attach to your actual seller is small — call it under ten in a hundred — and the entire value of an experienced examiner is telling which ten without ordering a certified copy of every one.

That is the job people now want to hand to an AI agent. It is also the job where being wrong is not an inconvenience, it is a claim. So before you let one anywhere near a live commitment, build the thing that decides whether it is good enough: a test set out of your own closed files.

What your examiner actually does with a name hit

Watch a good examiner for twenty minutes and you will see a sequence that almost nobody has written down. She reads the hit. She compares the middle initial and the suffix. She checks the address on the instrument against the chain on the tract. She looks at the date — was the judgment entered before or after your seller took title, because that decides whether it attached at all. She checks whether the docket shows a satisfaction. She checks the amount against how plausible it is for this property. If it is a federal tax lien she checks the refiling date and the redemption period. If it is a mechanic's lien she checks whether the statutory period has run. Then she writes two sentences in the file explaining her disposition, and those two sentences are the most valuable text your agency produces.

An agent evaluation in title work means running the agent over files whose outcome you already know, and comparing its disposition to the disposition your licensed examiner actually recorded — not asking it questions and admiring the answers. The difference between those two activities is the difference between a pilot and a demo.

The test set is 400 files you already own

You do not need to build data. You need to pull it. Take the last eighteen months of closed residential files and select roughly 400, weighted the way your actual order flow is weighted — if a fifth of your volume is refinance, a fifth of the test set is refinance. Then deliberately over-sample the ugly ones, because the average file teaches you nothing. Include:

• At least 30 files where a judgment or lien was a genuine match and had to be paid at closing.
• At least 30 with a common surname and a near-miss that was correctly cleared as not-our-party.
• Every file from the period where a claim was reported, a curative instrument had to be recorded after closing, or a policy was issued with a manuscript endorsement.
• Files with a foreclosure, probate or tax deed in the chain within the last twenty years.
• A handful of files where the search itself was wrong — the abstractor missed something — because you want to know whether the agent catches what a human missed, or copies the miss.

For each file, the answer key is what your examiner concluded and what ultimately appeared on Schedule B-I and B-II of the issued commitment, plus anything that had to be fixed afterward. That last column is the one that matters most.

flowchart TD
  A["Pull 400 closed files with examiner dispositions"] --> B["Agent runs each file in shadow, no output to customers"]
  B --> C["Compare agent disposition to examiner disposition"]
  C --> D{"Any missed true lien?"}
  D -->|Yes| E["Review the step record: which index, which instrument"]
  E --> B
  D -->|No| F["Widen: agent drafts B-II, examiner signs every file"]
  F --> G["Spot-check 10% monthly against the answer key"]

Watching the work, not just the answer

The thing that matured in 2026 is not the models — it is the ability to see, step by step, what the agent actually did to reach a conclusion. For a title agency that means a record you can open next to a file number and read: it pulled the grantor-grantee index for these years, it opened book 4127 page 812, it compared the middle initial, it treated the satisfaction recorded eleven months later as releasing the 2019 mortgage, and it decided the 2021 judgment against a John Miller with no middle initial and a different last-known address was not your seller.

That record is what turns a disagreement into a training conversation. When the agent and the examiner differ, you are not arguing about whether the machine is smart. You are looking at exactly which step it took wrong — and in practice you will find the errors cluster in three places: name variance (married names, Jr. and Sr., an anglicised spelling), timing (a lien that attached before or after the vesting deed), and satisfactions recorded in a different instrument type than usual for that county. Once you know the clusters, you know which files to route away from it entirely.

Scoring it: what a 3% miss rate actually means in claim dollars

Two numbers matter and they are not the same number. A false alarm — the agent flags something that is not your seller — costs an examiner ten minutes. A miss — a real lien it clears as not-our-party — costs you a policy claim. Score them separately and set your threshold on the second.

Read that table the right way round. It does not say the agent is bad; a 3.2% miss rate on lien matching is not embarrassing. It says that at your volume, 3.2% unsupervised costs about $47,500 a year, which is more than the examiner hour you were trying to save. So the agent runs as a first pass, the examiner reviews, and the saving comes from the 115 false alarms she now spends two minutes on instead of ten — because the step record shows her immediately why it flagged.

The three gates before it touches a live file

Widen its authority in stages, and write the promotion criteria down before you start so nobody negotiates them later under closing-week pressure.

• Gate one, shadow. It runs on live files but its output goes nowhere near the commitment. Every file is examined normally. You compare weekly. Minimum four weeks or 150 files.
• Gate two, drafting. It drafts the Schedule B-II exceptions and the B-I requirements and an examiner signs every single one. You are now measuring how much time the draft saves, and whether the examiner is actually reading or rubber-stamping — check by seeding a handful of files with a known planted discrepancy.
• Gate three, narrow autonomy. It clears only one category on its own — for example, name hits where the surname matches but the middle name, date of birth range and last-known address all differ, on files under your agency's ordinary liability limit. Everything else still routes to a person. Ten per cent spot check every month, forever.

What no test set will tell you

Your 400 files come from your counties, your search vendor and your last eighteen months of market. They will not tell you how the agent behaves in a county you just expanded into with a different indexing convention, or on a product you do not currently write much of, or during a refinance wave when volume triples and the pressure to skip the review is highest. Re-run the test set whenever any of those three change.

They also cannot tell you about the file where the right answer was not in any record — where an examiner picked up the phone, called the clerk in a rural county, and learned that the release had been recorded under the wrong instrument type in 2014 and the index was never corrected. That call has no digital trace and never enters your test set. It is also, on maybe one file a month, the entire reason a closing happens on time. Keep the person who knows to make it.

Frequently asked questions

Do we have to build the test set ourselves, or can the vendor supply one?

Build it yourself, and treat a vendor-supplied benchmark as marketing. The whole point is that it comes from your counties, your recording conventions, your underwriter's bulletins and your examiners' judgment. A test set from someone else's files measures somebody else's business.

Our examiner's notes are thin. Half the files just say "cleared."

Then your first project is not AI. Change the note template so every disposition records the reason in one line, and in ninety days you will have both a usable answer key and a materially better file for defending a claim. Agencies that have done this generally find the note discipline pays for itself before the agent ever arrives.

How long does this take before we see anything?

Assembling 400 files and their dispositions is a week of somebody's part-time attention if your closing software exports cleanly. Shadow running is four to six weeks. Call it a quarter before you are drafting, and do not compress it because a busy summer makes the saving look tempting.

Who inside a six-person agency owns this?

Your most experienced examiner, with your escrow manager as the second reader — not the person who is best with computers. The scarce skill here is knowing what the right answer was, not knowing how to run the tool.

One connected point. The same evaluate-before-you-trust logic applies to anything that speaks to your customers, including the phone. CallSphere builds AI voice and chat agents that answer title and escrow lines and web chat, take status questions, and book signings; the sensible way to bring one in is the same three gates — listen to real recordings against real calls first, widen only what it consistently gets right, and keep anything touching a commitment, a payoff figure or a disbursement with a licensed person.