Skip to main content

Voice: recognition, synthesis, interruptions

Module goal: set up the agent’s voice so that it hears the customer, sounds natural and stops talking at the right moment — in one language or in two within one call.
Voice is set up on the Settings tab of the editor’s left panel, in the Voice card; details are in Voice. The card has four groups: Speech-to-text · ASR model (STT, the STT parameters gear), Text-to-speech · Voice (TTS, the TTS parameters gear), Turn-taking (interruptions, active listening and background audio, the Turn-taking parameters gear) and the collapsible Audio input · advanced block. Turn detection is in the Additional window. The recognition model and the voice are selected from the platform directories; provider keys are neither stored nor entered in the agent. The languages the agent speaks are set not here but in the Agent languages field of the Agent & prompt card: recognition listens to all agent languages, and the voice speaks the current language of the call.
STT parameters gear: model and provider parameters

STT — what the agent hears: the model and provider parameters. The recognition model is picked from the directory.

STT parameters gear: model and provider parameters
Recognition hints. The model recognizes the company name, services, specialists’ names and terms more accurately if you list them in the keywords (comma-separated, weight — “word:1.5”) field of the STT parameters gear. The field is available for Deepgram, Cybernet, Soniox, Azure, AssemblyAI and Gladia models; a weight after a colon is understood by Deepgram. Cybernet models have their own rules:
  • the hints are sent to recognition only if the parameter passport of the selected model declares keywords; otherwise the pre-publish check warns that they are not sent;
  • a phrase is 3 to 80 characters long, at most 10 words, Cyrillic only, with no Latin letters, digits or duplicates; the service drops a phrase outside these rules, and the check warns about it;
  • more phrases than the model accepts is a check error: the service would refuse such a request entirely.
Hints matter most for vocabulary the model hears worse: names of transactions, products and cards, terms and abbreviations. List words and phrases separated by commas and write them the way they are pronounced. For Cybernet models this means:
  • Cyrillic letters, including Kazakh ones;
  • numbers spelled out as words rather than written as digits;
  • an abbreviation read as a word — in capitals;
  • an abbreviation read letter by letter — as the names of its letters, written out.
Case, stress marks, hyphens and punctuation don’t matter. The service drops a phrase with Latin letters, digits or an all-capital abbreviation without vowels. For example:
  • for a bank — chargeback, overdraft:2, Zelle, ACH transfer, routing number;
  • for a clinic — Medicaid, Dr. Ramirez, A1C test, endocrinologist.
Both examples are for a Deepgram model: overdraft:2 gives the word a weight of 2.
TTS parameters gear: output format and latency optimization

TTS — how the agent sounds. The gear holds the output format and latency optimization. The voice itself, its model and settings are picked in the voice window, and the reserve voice — in Additional → Reserve services.

TTS parameters gear: output format and latency optimization
The voice window. Click the voice in the Text-to-speech · Voice group, and a two-column window opens. On the left is the list: search, Platform voices / Own voices, provider tabs and the Model, Gender, Accent and ⚡ Low latency filters. The list holds only voices that speak every agent language. The ▶ button in a row plays a sample — a short greeting in the agent’s primary language. On the right is the panel of the selected voice: model cards, Speed, Fine-tune and Developer. Only the Use this voice button writes the choice; picking another voice resets the model and settings of the previous one. Details are in The voice window. Under Fine-tune the parameters are set with the fields of the voice’s passport: for ElevenLabs — Stability, Similarity and Style (0 to 1) and Speaker boost, for a cybernet voice — apply_text_normalization. The cabinet highlights a value outside the passport range and does not save it, and a parameter rejected by the passport blocks publishing. The passport of a cybernet voice is strict: it has no field for extra keys, and a key the passport does not know also blocks publishing — the service would silently accept a misspelt key. For a cybernet voice with a Speed of 0.9 or lower, the cabinet warns that the voice quality degrades.
Numbers, dates and abbreviationsHow the voice reads “12:30” or “$49.99” is decided by text normalization at the synthesis provider:
  • cybernet — apply_text_normalization with the values on and off. When the agent has a conversation language set, the service spells out numbers, abbreviations and Latin words only with on. If the agent says amounts, dates or times, choose on; keep in mind that on may add to the time to the first sound.
  • ElevenLabs — the same apply_text_normalization with the values auto, on and off; with auto the vendor decides.
  • Inworld — text_normalization with the values ON and OFF.
Tip: in fixed phrases it is more reliable to write numbers as words right away — “at twelve thirty” or «в двенадцать тридцать». Check the result with a test call.
Two languages in one callThe customer may start in one language and switch to another: in Kazakhstan — for example, from Russian to Kazakh (ru/kk), in the USA — from English to Spanish (en/es). To make this work:
  1. Agent languages — tick both languages in the Agent languages field of the Agent & prompt card. The first one is primary: the call starts in it. Keep Auto-switch to the caller’s language on — the agent switches to the language the customer speaks and answers in it; the voice, the recognition and the built-in phrases switch along.
  2. STT — a model that recognizes both languages. For kk/ru, suitable options are, for example, Soniox Real-time (it detects the language itself and switches within an utterance) and the Cybernet KZ/RU models; for en/es — Deepgram Nova-3, Soniox Real-time, AssemblyAI Universal-3.6 Pro (streaming), Gladia Solaria-1 (live) and Azure Speech STT. The picker itself keeps only the models that know both languages.
  3. TTS — a voice that speaks both languages. For kk/ru — cybernet and Soniox voices; for en/es — ElevenLabs, Cartesia, Inworld and Soniox voices.
  4. Tip: write fixed phrases in the primary language — the call starts in it.
For the kk/ru pair, the customer’s language is detected by the Kazakh letters in the utterance. A short Kazakh phrase without them, such as “salamatsyz ba”, keeps the call in Russian until the customer says a phrase with Kazakh letters.

VAD and interruptions

The Turn-taking group sets how the agent reacts when the customer starts talking over it:
  • Allow interruptions — the agent-wide switch; a node can set its own value.
  • Min duration and Min words — protection against false triggers: a cough, an “uh-huh” or line noise should not cut off the agent’s reply.
The sensitivity of the speech detector is in the Turn-taking parameters gear: the VAD runtime configuration table with the Activation, Exit and Silence thresholds, separately for the When the agent is speaking or thinking and All other states states, plus the endpointing delays and the Loudness gate.
Turn-taking parameters gear: end of turn, loudness gate and VAD rules

The Turn-taking parameters gear: endpointing delays, the loudness gate and the speech detector thresholds per agent state.

Turn-taking parameters gear: end of turn, loudness gate and VAD rules
An empty field means “auto”, and it does the math for youLeave Min duration and Min words empty — the platform fills in values for the recognition model you selected in STT. A streaming model gets 0.5 s and a minimum of 2 words, a batch model gets 0.6 s and 0. The difference is not cosmetic: a batch model (for example, Cybernet KZ/RU) returns the text in full once the customer has finished speaking, so a word count on it means “the agent stops not in the middle of the phrase but after it”.A minimum of 2 words is a deliberate trade-off. The agent is not interrupted by one- and two-word utterances: this protects against coughs and “uh-huh”, but “stop” and “no” are measured by the same yardstick. They are not lost: the short utterance is kept and joined with the customer’s next phrase — the agent reacts a beat later rather than mid-phrase.If the scenario requires cutting the agent off with a single word (for example, a refusal that must be caught immediately), set 0 — and keep in mind that a cough will then cut it off too. A number you enter always overrides “auto”.
Turn detection (end of turn) (Additional → Speech) determines how quickly the agent decides that “the customer has finished”. Too aggressive — the agent interrupts on its own; too cautious — awkward pauses. The mode depends on the agent’s language:
  • kk — VAD only: end of turn by silence; the cabinet recommends it when the language is outside detector support.
  • ru — Text model: the only mode that supports Russian; the model is marked deprecated.
  • en and es — Audio v1-mini: an audio model listens to intonation, rhythm and pauses and knows English and Spanish. Or STT provider, if the recognition model is Deepgram, Soniox, Azure or AssemblyAI: the provider determines the end of turn.
Background audio. Quiet noise is mixed into the agent’s voice: it does not reach recognition or the transcript. It is switched on in the Turn-taking group; in the Audio asset field you can take a standard platform background — Keyboard or Keyboard, short — or upload your own file. It gives the feel of a live call, but needs a careful volume. Details are in Background audio. Active listening (backchannel) — short acknowledgements during a long customer utterance. It is not a dialogue turn: the word goes on a separate audio track, does not reach the transcript and does not throw off turn detection. If the Words (comma-separated) field is empty, the words are taken by the current call language: for en — “uh-huh”, “mhmm”, “okay”, for es — “ajá”, “mhm”, “vale”, for ru — «угу», «ага», «мгм», for kk — «иә», «жарайды». Frequency (0..1) is 0.8 by default. Tip: 0.6 is a reasonable middle, 1.0 sounds intrusive. The moment for an acknowledgement is chosen most precisely with Deepgram, Soniox and Azure models; with the others, including the batch Cybernet models, the cabinet warns that the words may sound less natural. The synthesis cache serves already voiced phrases without synthesizing them again; it is on by default. For ElevenLabs, Cartesia, Inworld and cybernet voices, fixed phrases without variables are voiced in advance, at publishing, so the greeting plays without synthesis delay already in the first call; for Soniox and Azure the cache fills up during calls. Tip: Static instead of Prompt at the start of the conversation is not only about wording but also about speed.
A pause in speech — [pause N]You can put [pause 0.5] between words, and the agent stops for half a second; values from 0.1 to 3.0 are allowed. This is a way to read a phone number in groups rather than in one string, and to give the caller a moment before a question that requires a decision.The tag is understood both by the fixed text of a node and by generated replies — you can tell the model right in the prompt when to put a pause. The tag does not reach the transcript, History or reports: only synthesis sees it.But the pause length depends on the voice. ElevenLabs (except the eleven_v3, eleven_v3_conversational, eleven_v4 and eleven_v4_turbo models), Cartesia, Inworld and Azure hold a pause of exact length. Soniox, cybernet voices and ElevenLabs on these four models cannot, and there the tag turns into a comma or an ellipsis — the stop is audible, but [pause 2] and [pause 0.5] sound the same. A realtime agent has no pauses at all, and the validator warns about it. Check it by ear: make a test call, the transcript does not show pauses. Details are in phrase corrections and pauses.
The recognition model is selected in the directory window: search, the All providers filter and badges — Suggested, latency, No streaming. The voice is selected in the voice window: provider tabs, filters, the Recommended for this agent block and the No streaming and No word timing marks. Both windows show only the models and voices that support every agent language; an earlier choice that stopped fitting stays first, marked Does not support: ….