What's the hardest part of putting an AI voice agent on your phone lines?

Genuine question for everyone here, whether you went live already or stalled halfway. When we talk to small businesses, the technology is almost never the sticking point — it is everything around it. The answers we hear most often: trusting the agent not to embarrass the business on a live call, writing down the knowledge that lives in the owner's head ("we do accept walk-ins on Tuesdays, but only before 2pm"), and deciding when the agent should hand a caller to a human versus taking a message. Number porting and after-hours routing come up a lot too. So — what was it for you? If you have not launched yet, what is the thing holding you back? And if you have, what do you wish someone had told you before your first week of live calls? Honest answers help us decide what to build next, and they help the next business reading this thread.