Voice agent demos go well when the questions are predictable and the answers report fixed data values. Speech synthesis adds to the impression, since the voice is now realistic enough that few people in the room notice they’re listening to a machine.
But what if a caller needs the latest, but the information is always changing? An agent working from an inventory feed updated at midnight can tell someone phoning at noon that the headphones they want are in stock, hours after the last pair left the shelf. Callers have little reason to question it, since the voice sounds just as sure of itself as when the answer is right.
The agent’s speech on the call was clear and natural. Its mistake stemmed from the outdated inventory figure it read out. Hotel rates and train departure times go out of date just as quickly. On a phone line, the agent has well under a second to look up a figure like this, and there’s no screen to show a “last updated” note. If the lookup runs slowly or returns an empty result, the caller hears either an awkward silence or a guess.
Keeping Awkward Silences to a Minimum
In conversations between people, the gap between one speaker finishing and the next starting is on the order of 200 milliseconds, according to research by Stephen Levinson and Francisco Torreira at the Max Planck Institute for Psycholinguistics. Voice agents can’t match that yet, and every extra half-second of silence on a phone line reminds callers they’re talking to a machine.
In an example latency report in Deepgram’s documentation, recognizing the caller’s words and producing the language model’s first token take 460 ms. Generating the first audio brings the total to 640 ms, with no live data fetched yet. Every lookup against a product database or booking API adds to this delay. When the agent queries several sources at once, the slowest one decides how long the caller waits.
Some of the wait can be avoided by choosing carefully what the agent looks up during the call:
- Look up the most likely answers while the greeting plays.
- Check live only the figures that change often, such as prices and stock.
- Start the lookup while the agent is still composing its reply.
- Fill longer waits with a short phrase instead of silence.
Raw Pages Are Too Slow for a Phone Call
Not every figure lives in the company’s own database. Prices from suppliers, partners or marketplaces, and timetables from third-party sites, often have to be fetched from public web pages in real time.
Given a full HTML page, the agent has to dig through it to find the right price, which may be obscured by discount banners and prices for other models. Searching takes time the agent doesn’t have, and it might still settle on the wrong number.
Instead, discovery to parsing should be taken care of by the web access infrastructure. For the headphones question, the agent needs only two facts: whether the nearest store has the model in stock and its current price. Ideally these arrive as separate values the agent can read out directly, each with a timestamp.
Teach the Agent to Admit It Doesn’t Know
To stick with the headphones example, say the stock lookup times out mid-call. Unless it has been told otherwise, the agent may simply guess. It would be far better if it said it can’t check stock at the moment and offered a callback within the hour. Waiting for a call is a nuisance, but it beats driving across town for nothing.
Some figures go out of date much faster than others. Stock counts for popular headphones might be trustworthy for only a few minutes, while the store’s opening hours won’t change for months or years. The agent therefore needs a maximum age for each type of information, along with a rule for what to do once that limit has passed. For stock, it could try one more lookup and pass the call to store staff if that also fails.
Beyond Speech: Reviewing and Improving the Results
Pulled up the next morning, the log for the headphones call would show a quick reply to a clearly understood question. Unless the log was configured to record where each figure came from, it would provide no hint that the stock count was read from an inventory feed last updated at midnight. The retailer may then learn about the stale data only when the customer complains about a wasted trip to the store.
Listening back to recorded calls is the simplest way to find these mistakes. Calls where the customer complained or asked to speak to a member of staff are the best place to start, since a wrong answer is often the reason. Checked against its log, the headphones call would lead straight to the overnight inventory feed. Checking stock for best-selling products live at the moment of the call, rather than relying on a scheduled feed, would stop the same mistake happening again. Regular reviews also reveal questions the agent was unprepared for, such as whether a store can hold an item until the evening.
Speech quality will continue to improve with each new model, making demos sound better and better. On a real call, though, the agent will be judged by whether the product is on the shelf when the caller arrives.
