Your Support Bot Talks Over Customers. Native Voice Agents Don't

Old-school IVR meant pressing numbers into a phone tree. The first generation of conversational bots improved on that, barely, by stitching together speech-to-text, a text model, and text-to-speech, three separate systems handing work to each other in sequence. That chain added two to four seconds of dead air before every reply, stripped out tone and emotion in the transcription step, and fell apart completely the moment a caller interrupted mid-sentence.

Native audio-to-audio models skip the stitching. Speech goes in, speech comes out, through one integrated system, and the difference is immediately obvious the first time you talk to one.

Two Architectures, Side by Side

Stitched pipeline (real, noticeable lag):
[Audio in] → [Speech-to-text] → [Text context] → [Text model] → [Text-to-speech] → [Audio out]
                                                        │
                                    tone, cadence, and inflection get lost here

Native audio-to-audio (the actual improvement):
[Audio in / video feed] ─────────► [Realtime neural engine] ─────────► [Audio out]
                                          │
                        pitch, pauses, and natural interruption survive

Removing the transcription step doesn't just save time, it means the model can actually hear frustration, hesitation, a raised voice, and adjust its pacing in response, the way a person would. A transcript alone can't carry any of that.

What Actually Changes for Support

What Actually Needs Tuning If You're Building This

Real-time voice support over WebRTC has a handful of settings that determine whether it feels natural or actively annoying.

Turn detection needs a sensible threshold, too low and normal background office noise triggers false interruptions, too high and it stops responding to real speech. A moderate setting is usually the right starting point, then tune against your actual call environment.

Silence padding matters more than it sounds like it should. Set it too short and the agent barges in the moment a caller pauses mid-sentence to think, which reads as rude and cuts off information you needed.

And the agent should be able to call backend functions, looking up an account balance, checking a ticket status, without breaking the conversational flow. Returning that data quickly keeps the exchange feeling continuous instead of stalling on a lookup.

const sessionUpdate = {
  type: "session.update",
  session: {
    modalities: ["audio", "text"],
    instructions: "You are an enterprise technical support specialist. " +
      "When a user shares their screen, reference specific UI elements " +
      "by their label. Keep responses under two sentences unless " +
      "explaining a multi-step troubleshooting procedure.",
    voice: "alloy",
    input_audio_format: "pcm16",
    output_audio_format: "pcm16",
    turn_detection: {
      type: "server_vad",
      threshold: 0.5,
      prefix_padding_ms: 300,
      silence_duration_ms: 500,
    },
  },
};

Three Things That Will Actually Bite You

Audio tokens cost meaningfully more than text tokens, and an unbounded voice session adds up fast. Set a hard session timeout, ten minutes is a reasonable starting point, and route anything unresolved to a human rather than letting the meter run indefinitely.

If a caller is on speakerphone without headphones, the microphone picks up the assistant's own voice output and the system can end up interrupting itself in a feedback loop. Acoustic echo cancellation in the browser media stream isn't optional here, it's the difference between a usable deployment and an unusable one.

And low-quality microphones or loud background environments can cause the model to hallucinate words out of static, which in a support context means confidently acting on something the caller never actually said. A high-pass filter on the incoming audio, cutting anything below roughly 80 Hz, catches a lot of that before it ever reaches the model.

Frequently Asked Questions