Right article, wrong answer: where AI support agents break

AI support agents are giving wrong answers from correct knowledge base articles at a rate that knowledge base investment alone will not fix. A Columbia Journalism Review study of eight AI tools found errors on more than 60% of queries; a BBC and European Broadcasting Union study of four major platforms found roughly 45% of queries produced errors - with confident, unhedged responses even when the answer was wrong. The standard remediation, adding more documentation, addresses only retrieval failures. It does nothing for the larger and less visible category: generation failures, where the right source is retrieved and the wrong answer is synthesized from it.

What will matter most for AI support agent accuracy in the next 12-24 months?

The two dominant failure modes identified in this article - generation errors and chunk boundary failures - are not going away as models improve. Independent accuracy studies suggest a persistent error floor that knowledge base investment alone will not lower. Based on current evidence and deployment trends, three signals are worth watching.

Fact-accuracy error rates will stay stubbornly high

Even as retrieval systems improve, AI-generated answer accuracy on fact-verification tasks is likely to remain in the 40-60% error range through the forecast window. The BBC and European Broadcasting Union study, which found roughly 45% of queries produced errors across four major AI platforms, captures a persistent infrastructure problem rather than a temporary model immaturity. Josh Bersin, writing on the findings, noted that even a small error rate in training input can cascade across many queries, producing "ridiculous conclusions" from individually plausible steps. Support teams that expect the next model release to close this gap are likely to be disappointed. The practical implication: budget for human review layers as a permanent feature of AI support workflows, not as a transitional measure.

Action failures will grow faster than knowledge failures

The fastest-growing failure category in deployed support agents is not wrong answers - it is wrong actions reported as successful. As agents move from answering questions to taking actions (processing refunds, updating records, escalating tickets), the false-success failure type becomes more consequential than the wrong-answer type. A customer who receives a wrong answer can ask again. A customer whose refund was never processed but was told it was faces a different problem. The Cloud Security Alliance and Oasis Security found 78% of organizations had no policy for creating AI agent identities as of January 2026, meaning most deployments lack the access controls and audit trails needed to detect action-layer failures. This gap will close, but it requires investment in identity and delegation infrastructure that sits entirely outside the knowledge base.

Simpler, grounded systems will outperform complex agents on reliability

Cost pressure and reliability failures are already pushing buyers toward simpler, tightly-grounded chatbots rather than fully autonomous agents. Self-service AI support costs $0.10 to $0.70 per resolution when well-tuned, but "poor tuning increases escalations and repeat contacts, eroding savings," according to 2026 industry benchmarks. The practitioner experience of switching from an agent to a content-grounded chatbot, described earlier in this article, is a signal of a broader pattern: when reliability on edge cases is the primary constraint, less autonomous architecture often wins. Teams evaluating AI support options should weight edge-case reliability alongside deflection rate, and test generation accuracy explicitly before committing to an agentic implementation.

What are the two failure modes that cause AI agents to answer incorrectly?

The standard advice when an AI support agent gives wrong answers is: improve your knowledge base.

Add articles on uncovered topics, update outdated entries, expand the documentation for complex processes. That advice is correct for roughly one category of failures. For the other, it sends teams on months-long content projects that change nothing.

AI support failures divide into two structurally different categories, and only one of them responds to knowledge base improvements. Understanding which type you are facing determines whether you need more content or a different architecture.

Retrieval failure is the category the conventional fix addresses. The agent produces a wrong answer because it never retrieved relevant source material. The knowledge base either lacks coverage on the topic, or the retrieval pipeline is returning low-relevance results. Add the missing documentation, improve retrieval ranking, and the problem resolves. This is a genuine and common failure mode, and fixing it does require better content.

Generation failure is the category teams miss. Here, the agent retrieves the correct source article but still produces an incorrect or misleading answer. The knowledge base had the right information. The retrieval step worked as intended. The failure happened in what the model did with that information afterward: how it read a chunk of a longer document, how it synthesized multiple retrieved passages, how it compressed a nuanced conditional rule into a short response.

From the outside, all three look identical: the customer receives a wrong answer. The support team pulls the ticket and asks whether the knowledge base had relevant documentation. If the answer is yes - which it frequently is - the natural inference is to improve that documentation. That instinct is productive when the source material was genuinely ambiguous. It is wasted effort when the agent retrieved clear, accurate documentation and then mis-read it.

A 2025 Columbia Journalism Review study of eight AI tools found incorrect answers on more than 60% of queries. The BBC and European Broadcasting Union separately tested ChatGPT, Microsoft Copilot, Gemini, and Perplexity and found approximately 45% of queries produced errors, with tools presenting wrong answers "with alarming confidence, rarely using qualifying phrases." Those error rates persist even as knowledge bases grow, because a significant share of the failures are happening in the generation layer - not the content layer.

The misdiagnosis has a predictable signature. Teams update and expand the knowledge base. Retrieval logs confirm the relevant articles are being returned. The error rate on affected ticket types does not improve. That pattern does not mean the content work was wasted: it means the content work addressed the retrieval problem and exposed the generation problem that was always underneath it. Before any remediation project, the first question is not "what documentation is missing?" It is: "Is the right documentation already being retrieved when these failures occur?"

How does chunking turn a correct article into a wrong answer?

Most RAG-based AI support systems do not pass entire knowledge base articles to the language model.

They split documents into segments, called chunks, and retrieve the ones with the highest semantic similarity to the customer's query. Chunk size and segmentation logic are, in practice, among the most consequential variables in answer accuracy. They receive far less scrutiny than article quality.

The failure mechanism is straightforward. Consider a support article that reads: "Standard orders are eligible for a full refund within 30 days of purchase. Orders placed during promotional periods are eligible for store credit only and are not eligible for cash refunds." As a complete statement, the policy is unambiguous. Split the chunk boundary between the two sentences - which is exactly what fixed-length token chunking produces - and a customer asking "can I get a refund?" may receive only the first sentence as retrieved context. The agent answers yes, full refund, 30 days, with confidence. The correct answer depends entirely on when the order was placed.

This is not an edge case. It is the predictable outcome of applying fixed-length chunking to procedural content with conditional logic - which describes the majority of support documentation:

• Return and refund policies with date or tier-based eligibility conditions
• Pricing plans with feature gates, add-on exceptions, and legacy contract carve-outs
• Troubleshooting guides where a subsequent step depends on the result of the prior one
• Compliance content with jurisdiction-specific rules that override general policy

The agent's confidence does not reflect whether the retrieved chunk is complete. The language model generates a fluent, well-formed answer from whatever it receives. A fragment of a conditional rule produces a confident wrong answer for the same reason a complete rule produces a confident correct one: the model has no mechanism to detect that a crucial second clause was split off at the chunk boundary.

Patrick Dougherty, who spent a year building production AI agents over enterprise databases, documented that text-to-SQL accuracy caps at roughly 80% even on well-structured queries - partly because the agent-computer interface responds dramatically to small formatting and context changes. He found "huge fluctuations" in agent performance from seemingly minor tweaks, comparing the cascade effect to "a fender bender turning into a 30-car pileup." Support documentation has the same sensitivity. An agent interface built on fixed-length chunks of mixed content types will fail on conditional logic regardless of how accurate the source articles are.

Two architectural fixes address chunk boundary failures, and neither requires adding a single new article:

• Semantic chunking splits on paragraph and section boundaries rather than fixed token counts, preserving conditional logic that depends on adjacent sentences.
• Document-level retrieval returns full articles to the context window for shorter document types, eliminating boundary problems entirely at the cost of more tokens. Larger context windows have made this more practical in recent deployments.

If your failure analysis shows the right source articles appearing in retrieved context while answers remain wrong, investigate chunk boundaries before investigating content. A support article that reads correctly as a whole can produce systematically wrong answers when retrieved as a fragment.

How do you diagnose whether your AI agent has a retrieval problem or a generation problem?

The diagnostic step most teams skip is the one that determines whether any proposed fix will work.

Before launching a knowledge base expansion project or overhauling documentation, run a retrieval audit on a sample of failures. The audit takes less time than most content projects and produces a clear answer about where to invest, as of September 2026.

The process requires access to what the agent actually retrieved - the retrieved context that accompanied each failed answer. Most enterprise AI support platforms expose this in logging or debugging interfaces. If yours does not, that visibility gap is itself a problem worth solving before addressing accuracy, because you cannot fix a failure you cannot classify.

Pull 30 to 50 tickets where the agent gave a wrong or incomplete answer. For each, record what the agent retrieved and classify the failure into one of three categories:

1. Retrieval miss: The relevant documentation did not appear in retrieved context. The knowledge base may lack coverage, or the retrieval pipeline returned low-relevance results. Fix: content expansion and retrieval tuning.
2. Retrieval-correct, generation wrong: The right source appeared in retrieved context but the agent's answer contradicts, misrepresents, or oversimplifies the source material. Fix: prompt engineering, model evaluation, context window configuration.
3. Chunk boundary failure: A fragment of a relevant source appeared but without the conditional context required to produce a correct answer. Fix: semantic chunking or document-level retrieval.

The distribution across these categories determines where to invest. A majority of retrieval misses points to content gaps. A majority of generation failures points to architectural problems. A mixture of chunk boundary and generation errors, which is what I find most often in implementations that have already been through a content update cycle, points to the chunking and synthesis layer.

A fourth category is worth tracking separately: the false-success failure. A 2026 study of 11,755 AI agent runs documented cases where agents reported task completion when the task had not been completed. In one documented case, an airline agent told a customer a $686 refund had been processed when the database contained no record of it. The agent's response was confident and fluent. The customer had no way to know the information was wrong. Notably, five language-model judges scored worse than a coin flip at distinguishing false success from honest failure in that dataset - which means adding an AI reviewer does not solve this problem. It requires evidence-based verification, not evaluation.

One support practitioner who ran this kind of comparison directly described testing an AI agent that "broke half the time on edge cases" before switching to a simpler content-grounded chatbot inside Crisp. The simpler system, which retrieves and presents content without autonomous synthesis, proved "rock solid." That outcome is not an argument against AI agents in general; it is an argument for running the audit before committing to architectural complexity. The right level of complexity depends on which failure types your current implementation is producing.

Wade Foster, CEO of Zapier, has framed this well: "You probably don't want an agent. You want a workflow that thinks." For support teams, that distinction matters practically. The more autonomous the synthesis step, the more generation failures become possible. The audit tells you whether your current implementation needs more autonomy or less.

What should you do when your AI agent keeps getting it wrong?

The answer depends on which category of failure you are actually dealing with. That is the point this article is designed to make: before deciding on a fix, run the retrieval audit. Pull 30 to 50 failed tickets, examine what was retrieved, and classify each failure as a retrieval miss, a generation error, or a chunk boundary problem. The distribution tells you where to invest.

If retrieval misses dominate, expand the knowledge base and tune retrieval ranking. If generation failures and chunk boundary problems dominate, focus on chunking strategy, prompt engineering, and model configuration. If you see a mix, address the largest category first and rerun the audit after each change to measure whether the distribution shifts.

The broader lesson from independent AI accuracy studies is that error rates remain stubbornly high even at the model level, well above what most support teams expect when they deploy. The right posture is not to expect accuracy and investigate failures as exceptions. It is to design verification into the workflow from the start, so you have the data to distinguish one failure type from the other and fix each correctly.

I review AI support agent platforms with this framework in mind. If you want an independent assessment of how a specific platform handles retrieval, chunking, and generation accuracy before you commit to a deployment, Zaza Chat's AI support agent coverage is a good starting point.