About two years ago, I was building a voice-agent proof of concept orchestrated with n8n.
Whisper was the starting point. Speech came in, transcription turned it into text, an LLM and workflow decided what to do, tools were invoked when necessary, and the result was converted back into speech.
From a distance, it looked much like a modern voice agent. It could listen, decide, and answer.
Building it felt very different.
A surprising amount of the work was not “AI reasoning.” It was filling the gaps between components. When has the user finished speaking? How long should the system wait? What happens when speech recognition is wrong? What if the LLM is slow? What should happen if the user starts speaking while the system is still answering?
OpenAI introduced Whisper in September 2022. By 2023 and 2024, combining ASR, an LLM, and TTS made voice prototypes far more accessible than earlier telephony systems. But in my experience, the result was less like one Voice Model and more like a small distributed real-time pipeline.
I was building a voice agent, but most of what I was touching was orchestration.
Go back further and telephony itself was a specialist domain. ACD, IVR, CTI, and SIP belonged much more to contact-center and telecom engineering than to everyday application development. IVR had supported routing and self-service for decades, but many customer-facing automated experiences still felt menu-driven or constrained by directed dialogue.
As a user, the old callbot experience often felt simple: the human spoke naturally, while the system waited for one of the utterances it had been designed to expect.
That assumption has changed quickly.
Why Was a Voice Agent So Complicated Two Years Ago?
The architecture I was using can be simplified to:
Speech → STT → Text → LLM/Workflow → Tool → Text → TTS → Speech
It fits on one line. In production terms, every arrow is a failure boundary.
Slow STT delays everything downstream. An aggressive silence timeout cuts the user off; a conservative one makes the agent feel unresponsive. If the user speaks while a tool call is running, the application needs a policy for turn state. Each service brings its own latency, retry behavior, and error model.
That was one reason n8n was useful to me. The goal was not to invent a new voice model. It was to test, quickly, whether a business flow could be made to work by composing existing services.
The more the PoC worked, however, the clearer the architecture cost became.
The workflow had to understand too much about the modality itself: turn-taking, interruption, endpointing, and the timing between multiple AI services.
At that point, a voice agent was not only an AI application. It was also a small real-time system.
What Did the Realtime API and GPT-Live Actually Change?
The important change was not simply better speech recognition.
OpenAI introduced the Realtime API in October 2024, offering a low-latency speech-to-speech path without requiring developers to stitch together several models for every conversational turn. In September 2026, GPT-Live-1 arrived in the API with full-duplex listening and speaking, smoother interruption handling, and the ability to delegate deeper reasoning and actions to backend models, tools, or agent frameworks.
The generation shift, from a developer’s perspective, is close to this:
We used to assemble the conversation. The model and voice runtime are increasingly operating the conversation itself.
The live layer can absorb more responsibility for whether the user has stopped, whether they interrupted, and how listening and speaking overlap.
That does not make chained voice architectures obsolete. STT → LLM → TTS remains useful when transcript control, deterministic processing, or a particular combination of models matters. What changed is the architectural choice.
The first question used to be, “Which STT and TTS should I connect?”
Now a more useful one is, “How much of the live conversation should the voice runtime own, and where should it hand control to the backend agent and business systems?”
What Is Actually Hard About an Enterprise Voice Agent?
Once the model can speak naturally, a useful thing happens: the Enterprise problems become easier to see.
The system still has to know who the customer is. It needs the relevant case, order, account, or campaign context. It needs to know when authentication is required. It must distinguish between answering from knowledge and performing a business mutation. It needs to pass context to a human when escalation is appropriate. When the call ends, the outcome has to survive in a system of record.
Outbound adds another layer. Is this customer eligible to be called? Has the operation already succeeded? If the provider result is ambiguous, is it safe to retry? Was that voicemail, a short human hangup, or a failure before the conversation actually started?
A natural-sounding model does not resolve those questions.
In Deloitte’s 2026 research on US banking contact centers, 77% of surveyed executives cited integration of new technology with other systems and tools as a major modernization challenge. That is a banking-specific sample, not a universal industry statistic, but the architecture problem matches what I have found in practice.
Conversation is something a Voice Agent can own. There is no reason to hand it the entire business lifecycle.
I usually separate at least these responsibilities:
- Conversation — listening, speaking, interruption, and live interaction
- Telephony — PSTN/SIP, phone numbers, and call transport
- Business Context — CRM customer, case, opportunity, or campaign state
- Tool Execution — what may be read and what mutations are allowed
- Durable State — canonical business facts that must survive after the call
- Authority — what the AI may complete autonomously and when it must escalate
One vendor can own several of these layers, or an enterprise can split them across systems.
The important part is knowing which decision you are making.
Where Does a Voice Agent Actually Sit in Enterprise Operations?
I find three patterns useful.
The first is inbound self-service. The customer calls, the agent identifies the intent, retrieves account or order information, performs bounded work, and escalates to a person when it lacks authority or sufficient context.
The second is outbound engagement. A CRM or campaign signal initiates the operation, and the voice agent contacts the customer for qualification, confirmation, reminders, follow-up, or another bounded task.
The third is human agent assist. The human remains in the conversation while AI provides transcription, retrieval, suggested responses, next-best actions, or post-call summaries.
The same voice model could participate in all three.
Their operating contracts are not the same.
Inbound often emphasizes authentication and escalation. Outbound makes eligibility, calling policy, retry semantics, and durable CRM outcomes much more important. Agent assist leaves the final utterance and business decision with the human.
The Voice AI can be the same while the authority model is completely different.
How Are CRM-Native, CCaaS, CPaaS, and Realtime APIs Different?
Voice-agent comparisons quickly turn into long feature matrices.
I find it more useful to ask how much of the stack each approach expects the vendor to own.
A CRM-native approach begins close to customer context and business actions. It is attractive when the CRM is already the center of identity and workflow.
CCaaS expands the boundary to queues, routing, recording, workforce, analytics, self-service, and the human-agent desktop.
CPaaS provides programmable communication primitives such as phone numbers, call control, and media transport, leaving more agent and business logic to the application.
A realtime model API provides the intelligence and live audio layer with substantial flexibility, but the application still owns much of the session lifecycle, telephony integration, and durable business state.
A specialist voice runtime such as Ultravox sits somewhere in between: it packages voice-first agent runtime capabilities, telephony integration, tools, and grounding without requiring the enterprise to replace its full contact-center estate.
The useful question is not, “Which product wins?”
It is:
Which layers do we want a vendor to own, and which authority and state must remain ours?
Once that is explicit, product selection becomes much easier.
Why I Found Ultravox Interesting
I did not start testing Ultravox because I had concluded that it was the universally best Voice AI.
I already had Salesforce. Salesforce already owned customer and campaign business state. I could choose a telephony provider separately. I did not want to replace all of that.
The layer I wanted to change was the Voice Runtime between them.
Ultravox describes its telephony model as BYOT — Bring Your Own Telephony and currently supports providers including Twilio, Telnyx, Plivo, and Exotel as well as SIP. Its runtime includes inbound and outbound calling, transfer, DTMF, and other voice-specific primitives.
That architectural fit mattered more to me than a benchmark headline.
The CRM could remain the CRM. Telephony could remain telephony. I could insert or replace the layer that handled the conversation.
That is also how I approached an outbound pilot. Salesforce remained the owner of customer and campaign state. The voice runtime concentrated on the live conversation. The telephony provider carried the call.
The internal object model, reconciliation, and retry details belong in a deeper implementation article. The point here is the responsibility boundary, not the provider names.
This is not an Ultravox-only answer. A team can wrap a realtime API directly or choose a CRM-native or CCaaS architecture that owns more of the stack.
But when the goal is to preserve existing business systems while changing the voice layer quickly, a specialist runtime is a practical option.
Is Human Handoff a Failure of the Voice Agent?
Voice demos naturally encourage the question, “How far can this work without a person?”
Production architecture needs a different question.
In Gartner’s 2026 customer survey, 87% of respondents said access to a human agent was essential when companies use GenAI for customer service.
As with any survey, that result should not be treated as a universal law. But it is a useful design signal.
If AI-to-human transfer is modeled only as a failure, the automation will try to retain work longer than it should.
If it is a normal state transition, the architecture becomes clearer.
The voice agent handles the interaction while it has enough context and authority. If identity verification, ambiguity, emotion, exceptions, or high-consequence decisions require a different authority, it transfers the interaction — along with the context already collected.
Good automation is not automation that handles everything alone. It is automation that knows when it no longer has the authority to continue.
That principle is not unique to voice.
Phone calls simply make the boundary more visible because the interaction is synchronous and a bad transition cannot be repaired quietly later.
Voice Is a Strange Interface
There is an interesting pattern in technologies such as GPT-Live.
Look at the problems modern voice models work hard to solve and many of them are properties of human conversation.
People interrupt. They hesitate. They use “uh” to hold a turn. They respond before the other person has fully stopped. Reply too quickly and it feels rude; reply too slowly and it feels as if the connection is broken. Tone and cadence carry information that is not in the transcript.
To make AI better at phone calls, we are teaching it to handle these very human imperfections.
APIs do not have most of these problems.
An API does not wait to see whether the other endpoint is taking a breath. A JSON payload does not feel awkward and hesitate. One service does not accidentally talk over another endpoint.
Voice agents look futuristic, but they are also deeply human-shaped technology because they assume a human communication interface on at least one side.
If Both Sides Are Agents, Why Make a Phone Call?
After building enough of this stack, a slightly cynical question appears.
Today, the default picture is:
AI Agent → Voice/Phone → Human
In the future, a person may put their own agent in front of the call as well:
Human → Personal Agent → Voice/Phone → Enterprise Agent → Business System
What happens when the actual conversational actors on both sides are agents?
Does one AI really need to generate a sentence, synthesize speech, send it through the telephone network, and have another AI recognize the speech and reconstruct the meaning?
If both sides are machines, a structured request, API, or MCP-style contract may be faster, cheaper, and less ambiguous.
That does not mean telephony is about to disappear.
Phone numbers, contact-center infrastructure, CRM processes, regulation, and customer habits will persist. More importantly, there will continue to be a human on at least one side of many calls for a long time.
That is why I increasingly see the Voice Agent not as a technology for a world after humans, but as an interface for a long transition in which humans and AI share the same business systems.
Two years ago, the PoC challenge was making a set of APIs behave enough like a conversation to be useful.
Now the conversation itself has become remarkably capable, and the engineering attention moves outward — toward identity, authority, state, and integration.
And if the day comes when there is an agent on both ends of the phone, a different question may become more interesting than how human the conversation sounds:
If both sides are agents, why are they making a phone call at all?