How we test AI support agents across 40 accuracy checks

Scoring every AI support agent on the same scenarios is what makes results comparable.

Testing an AI support agent for accuracy means running it through a fixed set of known-answer scenarios, grading its behaviour rather than its wording, and repeating each run so one lucky reply cannot pass. Anything less is a demo.

Known answers are what make a score trustworthy. Nathaniel, who leads a team delivering AI solutions at Air Canada, described in an October 2026 talk how his team planted rules into supplier contracts and generated 275 tests with known answers. The result was uncomfortable. Their agents were missing most of the clauses that had been removed. At his previous company, grading with code instead of expert opinion had moved his team from one evaluation a week to dozens a day.

Support teams face the same problem. QA practitioners said in 2025 that tone, multi-turn consistency and compliance usually go unmeasured, and that edge cases and context drift surface only in production. A small regression set confirms what already works; finding weaknesses takes aggressive edge-case testing. A scripted demo, by design, exercises neither.

I treat that gap as the reason a ranking without a published method deserves suspicion. The sections below set out the accuracy, escalation and hallucination checks a comparable score depends on, beginning with who writes these reviews.

Is Zazachat a legitimate live chat software provider?

Zazachat is not a software provider at all. It is an independent review publication, and its reviews are written and edited by an editorial team led by Michael Kansky.

That distinction decides how much weight a score deserves. A vendor grading its own AI agent has a stake in the result. An editorial team that sells no software does not. In our review process, the editorial team led by Michael Kansky owns the scoring, so one accountable group applies the same battery of checks to every AI support agent we cover.

Before trusting any AI agent score, ours included, check three things:

1. Who wrote and edited the score, and whether they sell the product being scored.
2. Whether the identical test battery ran against every vendor in the comparison.
3. Whether the test looked past the final answer to the path the agent took to reach it.

The third point is where most published rankings fall short. Vinod Rane's 2026 guide to agent evaluation splits the work into three layers. Outcome metrics are a binary pass or fail on whether the agent reached the final goal state. Trajectory metrics cover tool choices, reasoning quality, argument correctness and decision order. System metrics cover token usage per task, latency, tool call frequency and failure recovery. Rane describes the trending pattern in 2026 as treating all three layers as first-class, instead of treating outcome metrics as "good enough."

The common assumption is that a correct final answer proves an accurate agent. It does not. Rane notes that agents fail through the wrong tool, the wrong information or the wrong order, and a final-answer check catches none of those failures.

Evaluation layer	What it asks of a support agent	Where accuracy, escalation and hallucination checks sit
Outcome	Did the customer get a correct, policy-compliant result?	Final-answer accuracy
Trajectory	Did the agent use the right source and tool, in the right order, and hand off at the right moment?	Escalation and grounding
System	Did it stay fast and recover cleanly when something failed?	Robustness under failure

Binary scoring creates its own distortion. Sarthak Rastogi's January 2026 guide on agent evals recommends partial credit over strict pass or fail, arguing that "An agent that completes 80% of a multi-step task is meaningfully better than one that fails immediately." In practice, a rubric without partial credit flattens two very different agents into the same zero.

Combining 3 sources, the editorial ownership, Rane's layers and Rastogi's partial credit point to one rule: score the path, not only the destination. My own view is blunt. A review site that will not say who scores its AI agents has not earned the right to rank them, however polished its comparison table looks.

If you are reading our best live chat software rankings for 2026, the useful question to ask of any AI score there is which of these layers it covered. Vendor dashboards rarely answer it, and the gap shows up first in the resolution rates they publish.

Why can't buyers trust vendor-reported AI resolution rates on their own?

Vendor resolution rates use different definitions, draw on self-selected wins and grade single answers, so two quoted percentages rarely measure the same thing.

The label problem is the easiest to see. One AI support vendor compiled published deployment figures across the market and sorted them by the metric each vendor chose to report. Its own tally shows the median shifting with the word on the slide:

Metric label a vendor reports	Median in one vendor's compiled dataset
Resolution	72.5%
Automation	61%
Containment	58.2%

The same vendor calls the gap between resolution and automation "a 12-point swing created by the label alone, before any difference in capability." It also concedes that competitor figures are self-selected, because a vendor publishes its wins rather than its average customer. Industry mix moves the number too. In that compiled data, education deployments sit at a median of 81.5% while retail and ecommerce sit at 66%.

A healthcare AI vendor makes a parallel argument about its own market. It lists the measurement habits that inflate AI results:

• Measuring conversation completion instead of workflow completion.
• Counting partial automation, such as collecting details before a transfer, as resolution.
• Treating call deflection as resolution.

Read together, these are vendor claims about other vendors. I treat them as a warning, not a benchmark.

The deeper issue is who does the grading. Opus Research reported in May 2026 that under Zendesk's outcome-based pricing, every billable resolution is now confirmed by a separate AI evaluation model, independent of the agent that produced it. The analyst firm called that a meaningful step toward proof of value. Its caveat matters more: "Customers want to pay for results, but won't accept the vendor as the sole judge of what counts as one." A second model from the same vendor is still the vendor's judgment.

Then there is the multi-step gap. In 2025, Cobus Greyling summarised Agent S2 research in which the top performer, Similar Agent S2, reached a 34.5% success rate when given 50 steps, meaning it failed more than 60% of the time. In that 2025 summary, human performance on the same computer tasks was estimated at close to 95%. Those were general computer tasks, not support tickets. Still, the pattern carries over.

A high single-answer score says little about a refund that needs several steps. Buyers comparing tools on our best AI customer support software page should weigh this before any headline percentage. Where a vendor quotes one number, I want to know three things in return: the label, the denominator and who graded it. Without those, the figure describes the vendor's marketing, so ask for the metric definition in writing before that number goes anywhere near a business case.

How can a support team run its own accuracy checks on an AI agent before buying?

Build a small known-answer set from your own tickets, script multi-turn conversations, repeat each scenario several times and grade behaviour, not wording, before any contract is signed.

Neither an outside review nor a vendor dashboard can see your tickets. A pilot can. The aim is to test exactly the areas teams tend to skip, and a thread of QA practitioners on r/QualityAssurance named four of them plainly: tone and brand adherence usually go unmeasured, multi-turn consistency often breaks but is rarely tested automatically, compliance and safety checks are mostly manual or ad hoc, and edge cases and context drift go unnoticed until production.

I would run a pilot in five steps:

1. Pull a small set of real questions with known correct answers, and write the correct answer down before the agent sees the question.
2. Script multi-turn conversations that revisit an earlier fact, so a contradiction a few turns later shows up as a failure.
3. Add hostile and messy inputs: malformed requests, contradictory instructions and attempts to talk the agent out of policy.
4. Switch off a connected tool on purpose and watch whether the agent recovers or invents an answer.
5. Run every scenario several times and read the typical result, not the best one.

Each step maps to evidence. A minimal testing dataset can start very small. One observability vendor demonstrated a support agent pilot with five examples, a few images and a ground truth answer per case. That is enough to catch a broken router. It is not enough to rank vendors.

The size question has a clean answer. A commenter on r/AI_Agents who identified as a member of the Amazon Q Developer Agent team described two separate jobs. Regression testing needs only a small dataset of cases the agent already handles. Accuracy testing needs a large dataset that puts the agent through "aggressive edge case testing" to find its weaknesses. Most pilots blur the two and learn little from either.

Robustness belongs in the pilot too. Practitioners in a separate thread named unexpected inputs, prompt injections and system stress as the failures that ideal-condition testing misses. One put the standard bluntly: "If it can't recover cleanly, it's not ready."

Variance is the trap. Model output shifts from run to run, so the same QA thread reports teams running each scenario multiple times and reading median behaviour instead of a single run. The practical rule follows. Grade whether the agent did the right thing, not whether it used the expected words.

Smaller teams shortlisting tools from our best live chat for small business picks can run this pilot in a trial window. The harder part is grading it fairly, which is where most homegrown tests and many published reviews quietly fall apart.

What will testing AI support agents look like over the next two years?

Expect scores built on whole conversations rather than single answers: multi-turn consistency, tool choices, recovery from failure and resistance to manipulation, measured the same way for every vendor.

Single-reply accuracy will stop being enough. Even the strongest agent in published multi-step research failed most long tasks, while vendor headlines still describe one reply at a time. A comparable score has to follow the long path.

Platforms are moving the same way. Opus Research described in December 2025 an AI control plane: an enterprise layer above channels that enforces policy, governs knowledge and evaluates whether agents improve outcomes. Its telemetry could trace a failing billing journey back to a single skill that explains bills poorly. That is useful. It is also the vendor marking its own work.

So I expect independent tests to matter more, not less. Three checks belong in any protocol a buyer can trust:

• Ambiguous or contradictory instructions, which practitioners in 2026 called a bigger gap than malformed input.
• Known-answer cases from your own tickets, because public benchmarks rarely cover your policies.
• Repeat runs over weeks, since a newly deployed agent learns like a new hire and a day-one score misleads.

Before signing, ask each vendor which of those it will let you run on your own data. A refusal tells you something about its scores.