Skip to main content

Testing an agent

The Test button in the editor toolbar opens the test panel. It has two tabs:
  • Voice / Chat — you talk to the agent yourself. The tab is named after the open mode: Test opens a call — the Voice tab — and its menu switches to Chat and back; see Test before launch.
  • Scenarios — the platform talks to the agent for you, following scenarios you describe in advance, and tells you whether the agent is ready to launch. This page is about this tab.
A check is a batch of test conversations with the agent. Test callers talk to it following the scenarios. After each conversation a judge decides whether the agent did what was expected, and at the end you get a result: whether the agent is ready to launch, and what to improve. The check makes no real calls and writes nothing to History — the conversations stay in the check result.

How it works

  1. Scenarios. You describe who calls and what you expect from the agent, or let the platform generate scenarios from the agent settings.
  2. Check. Test callers talk to the agent following every enabled scenario, in parallel.
  3. Judge. After each conversation a judge model compares it with the expectation and marks the line where the agent went wrong.
  4. Result. Ready to launch, Not ready or Not everything was checked — by the thresholds of the organization.
  5. What to improve. The platform groups the failures and suggests changes to the agent.
  6. Fix with Copilot. Copilot proposes a change, you apply it, and the scenario is checked again.
  7. Publishing. The publish dialog shows whether the check passed.

Scenario

A scenario describes one conversation. It is stored with the agent and outlives the checks: the same set can be run after every change. Manners: Ordinary; In a hurry, irritated; Rude, threatening; Confused; Elderly, asks again; Answers in one word; Slang, typos. Marks of a scenario in the list: Attack — an attack scenario, Own — added by hand, No knowledge base — “With the agent’s knowledge base” is unticked. An attack is a scenario like any other: you can turn it on and off, edit and delete it, and have as many of your own as you like. The checkbox turns a scenario on and off: a disabled scenario stays with the agent but does not run. Enabled scenarios come first in the list, disabled ones last. The first 10 rows are shown; Show more (N) opens the next ones. Clear selection in the list header unticks every scenario; when none is ticked, it turns into Select all.

Where scenarios come from

  • Your own — the Add your own scenario button.
  • Generation — Generate from the agent settings. Choose how many with − 8 +, from 3 to 20. With with the knowledge base ticked, callers will ask about facts from the agent’s knowledge bases. Scenarios appear in the list as they are ready; Cancel stops the generation and keeps what was already created. More adds scenarios without repeating the existing ones and asks for the number and the knowledge base the same way. If the agent has no attack yet, one of the ordered scenarios will be an attack: order 3 and you get 3, one of them marked “Attack”. Scenarios are written in your cabinet language; lines the caller must say word for word stay in the agent’s language.
Scenarios in another language. A scenario written in a cabinet with another language is shown to you translated into your cabinet language — in the list and in the form. The model translates it once, and the translation is kept for everyone with that cabinet language; quoted words, names, numbers and variables are not translated. The form marks such text Auto-translated from …, and Show original shows the source text. The check and the judge always work on the original, so a translation cannot change the result. If you edit a scenario’s text and save it, your version becomes the new original and the old translations are dropped.
  • Copilot — ask in the chat: “add a scenario where the caller…”. Copilot saves it with the same fields.
  • From coverage — after a check: Create scenarios for these nodes (see below).
If the agent has no enabled scenario, the check runs five base callers: calm, in a hurry, confused, off-topic and attacking. Base callers are not saved with the agent. If there are enabled scenarios, exactly those run — nothing is added beyond what is ticked. Delete all scenarios in the list header removes the agent’s check scenarios. Past check results stay.

The check

The Check the agent button in the tab footer shows how many scenarios will run — the number of ticked rows. With every row ticked it matches the number in the list header.
  • What is checked — the draft or a published version. The choice is the chat test’s: What to test in the test configuration on the Chat tab (the summary button, for example Draft · rigid · mock). A draft with schema errors cannot be checked: the button stays disabled until they are fixed.
  • A conversation — up to six turns; the agent speaks first. Conversations run in parallel, and the progress bar shows “Conversations in progress · N of M”. Stop interrupts the check: no new conversations start, and the ones in progress finish their current turn — meanwhile the tab shows Stopping the check and how many conversations are still running. Scenarios that never started get Not run. The finished conversations of a stopped check open from its result, with ← History above it (or ← To scenarios if there are no past checks).
  • Agent functions answer with test stubs: the check calls nothing outside. The judge knows this and judges the logic of the step, not specific values. Functions without a test answer are listed before the start.
  • The agent model is unavailable — the check does not start, and the tab says so: “The agent model is unavailable: … Check the LLM settings of the agent”. If the model fails in the middle of a check, the check stops with the same reason instead of marking every scenario separately.
  • One check at a time per agent: while one runs, a second cannot start.

How the judge decides

The judge reads the whole conversation once, at the end. The scenario’s expectation and the agent’s rules are its criteria, and it decides for each one whether it was met. A scenario passes when no criterion is broken. In the conversation card the judge explains the decision in the cabinet language and marks the line where the agent went wrong — Error here. If the judge skipped a criterion, it is asked once more. If it still does not rate it, the criterion does not count as a failure, and the conversation card shows The judge did not rate: …. If the judge could not answer or the conversation did not happen, the scenario gets could not run. This is not an agent error: such a scenario does not count as a failure; repeat the check.

The result

The result line reads like this: “Not ready · 3 of 5” — 3 conversations of 5 failed; with Ready to launch the number counts the passed ones. Attacks are counted along with the other scenarios. Expand the line to see the thresholds and the number of breaches. Thresholds are set per organization: at least 95% of scenarios must pass, with 0 breaches. A spot check — Check again on one scenario, or a run from Copilot. It updates that scenario’s outcome but gives no “ready to launch” result: that needs a full check. The result is outdated — the agent changed after the check: the draft was edited or a new version was published. Check again before publishing.

What else the result shows

  • The Scenarios and What to improve tabs. Failed scenarios come first, then those not run, then other outcomes, and the passed ones last.
  • Knowledge base — a mark on a scenario where the agent answered from the knowledge base; in the conversation card such a line is labelled Answered from the knowledge base.
  • Comparison marks against the last full check: Broke — passed before, fails now; Fixed — the other way round; New; not checked; could not run.
  • Node coverage: “The conversations did not reach N nodes of M” and the first names of those nodes. The check does not know what the agent says in nodes it never reached. Create scenarios for these nodes adds scenarios that lead there.
  • The conversation card: the whole dialogue, the judge’s verdict, “Expected from the agent”, the variables of this conversation and the Error here mark. The node with the error is highlighted on the canvas.

What to improve

After a full check with failures, the platform groups the errors and, for each group, writes what is wrong, why, in which scenarios, how to fix it and where — the general prompt, a specific node or the settings. Recommendations are prepared separately from the result: the result is shown at once, with “Preparing recommendations…” next to it. If they did not make it in time, Get recommendations asks again.

Fix with Copilot

Fix with Copilot on a failure or on a recommendation group opens Copilot with the conversation from the check attached and a ready question. Copilot proposes a change to the agent; a person applies it with Apply. Then “The change was applied to the draft” and Check the scenario again appear. After the fix the conversation card shows What changed in the agent and Before → After.

Publishing

The publish dialog shows two rows:
  • Schema — schema errors block publishing.
  • Check — passed; failed (with Open the results); not run (with Check the agent); the agent changed after the check (with Check again).
The check does not block publishing: if it failed, the button becomes Publish anyway.

Check history

If the agent has been checked before, the tab opens on the scenario set with the Past checks block above it: date, version, result and number of scenarios. The last three are shown, the rest under Show all (N). A row opens its result; ← History returns to the list.

Who can run checks

The Agents and editor section is open to the administrator and the developer, so the check is theirs only (see Roles).

Good to know

  • Check conversations are test conversations: they do not enter History or call analytics.
  • The check runs with test answers of functions: the agent’s real integrations are not called.