Agents can meet users through several different surfaces: a chat window, a voice call, or no conversation at all — an event trigger that runs in the background. These aren't stages in a progression. They're distinct design choices, each with its own constraints, and the choice propagates into the rest of the architecture. This is a guide to making it deliberately.
The same agent, three ways
It helps to hold one job fixed and vary the interface. Take an agent that receives a maintenance request from a tenant, checks the property record, judges urgency, and either schedules a contractor or escalates to a human.
Chat. The latency budget is generous — a few seconds of processing reads as normal. Output can be structured: a list of time slots, a confirmation card, a link. Errors are recoverable because the user can scroll back and see what was proposed. The transcript doubles as an audit trail. The cost is that the user has to come to the interface.
Voice. The latency budget tightens to around a second before silence becomes awkward, which constrains how much reasoning can happen between turns. Structured output isn't available — six time slots read aloud is an interrogation, not a choice. There's no scrollback, so confirmation has to be built into the conversational flow rather than added as a UI element. What you get in return is immediacy and accessibility: no app to open, no typing.
Ambient or event-triggered. No turn-taking to design at all. The agent acts on an incoming email or a sensor reading, and the human sees the result afterward. The design work moves from conversation flow to reporting and guardrails: what the human reviews, how they audit a decision already made, and which actions are irreversible enough to require a stop.
Same job description, three different systems — different tool schemas, different memory requirements, different human-intervention points.
Questions that settle the choice
Where is the user when the need arises? Someone with a burst pipe has a phone in hand and water on the floor. Someone reviewing the week's escalations is at a desk with no time pressure. This tends to be the strongest single signal.
Does the output need structure? If the useful answer is a table, a comparison, or a set of options, a screen carries it and speech doesn't. If the answer is a short confirmation or a single next step, voice handles it fine.
How expensive is a misunderstanding? Cheap-to-correct interactions tolerate voice. Where an error is costly or hard to detect, the scrollback and explicit confirmation of a screen are worth the friction.
Is a human present at the moment of action? If not, you're building ambient, and the design budget shifts toward reversibility and reporting.
What's the accessibility profile? Voice serves users who can't or won't type; screens serve noisy environments and users who need to review before committing.
What each choice implies for the stack
Once the surface is settled, much of the architecture follows.
Voice points toward speech-to-speech models rather than the older transcribe-reason-synthesize pipeline, which compounds latency at each hop and discards prosody. Tools like ElevenLabs operate at this layer, alongside real-time voice APIs from the major model providers. Underneath sits transport — WebRTC, interruption handling, turn detection — which is what LiveKit and similar services exist to handle. The tool schema should return one option at a time, and there needs to be an escalation path that hands a human the transcript.
Chat opens the door to generative UI, where the agent returns renderable components instead of prose. This is where the modality earns its keep: an interactive form or chart uses the screen for what it's good at.
Ambient leans on orchestration and observability — queues, retries, traces, and the ability to reconstruct why the agent acted as it did.
When you need more than one
Most production agents eventually span surfaces: take the call, send a text confirmation, post a summary to a queue.
The workable pattern is one core with several interaction surfaces, with state held in the core rather than in any surface. If conversation history lives inside the chat interface, the voice path can't see it. If escalation logic is written into the voice flow, the ambient path can't reuse it. Deciding which surface is primary — and building the others as additional entry points to the same core — keeps this from becoming three agents maintained in parallel.
A practical sequence
- Identify the user's context at the moment of need.
- List what the modality costs you in that context, not just what it gives you.
- Decide where the human confirms or intervenes.
- Pick the stack, which by this point is largely implied.
- If multiple surfaces are needed, choose the primary one and design the shared core around it.
The main thing this ordering avoids is picking tools before the question they answer is clear.
Your agent starts with a blueprint. Build yours free →