Four of 68 is a content failure, not proof that AI chat doesn't work
In November 2026, CustomerThink published a test by Luke Wilcox. An AI agent posed as an anonymous visitor on public company websites and asked each chatbot three questions. They were: what are your customer service hours, how do I update billing or payment details, and how much does the main product cost. Only 4 of 68 bots answered all three on the first attempt. Wilcox started with 2,764 US-listed companies (NYSE or Nasdaq, filing a 10-K, $50M+ 2025 revenue). He drew a random sample of 120 and found a chat he could open at 102. Of those, 68 opened with a bot.
How strict was the test, and who set the bar?
Wilcox, the author, set it. Each reply was graded full, partial or no answer. Generic links, contact-us messages, wrong answers and silence all counted as no answer.
Per CustomerThink, "full" meant a direct, specific answer inside the chat. For billing, that meant steps or the place to make the change. A short answer plus a link to the right page counted as partial. The excerpt we saw cuts off before the results are broken down, so we can't say how many bots earned partial credit or which of the three questions failed most often. The headline figure is also ambiguous about whether partial answers counted towards a pass. Ask for the breakdown before quoting the number.
What does the sample actually represent?
Only bots that greeted an anonymous visitor without asking for details. Of 120 sampled companies, 102 had a findable chat, and 68 of those opened with a bot.
The rest of the 102 demanded a name, email, phone or account number, or opened with a human. Those were left out. If your bot sits behind a login or a lead form, this test says little about it. The sample is also large listed companies, not small businesses, so the result may not carry over to a mid-market buyer's vendor.
One angle the write-up doesn't dwell on: hours, billing steps and price are static facts a company can publish once. A bot that fails all three usually has a gap in its knowledge base or a deliberate deflection rule. That is a configuration problem, not a limit of the model.
Can we trust an AI agent as the grader?
Partly. A second agent agreed with the first on individual questions 85% of the time and on the overall verdict 100% of the time. That check covered only a small re-run.
An 85% match means some individual calls came out differently the second time. Both graders were agents, so this shows consistency, not independent verification. Wilcox also ran a small set by hand in Chrome, but the result is cut off in the text we had. The price question is a further weak spot. Many enterprise vendors don't publish prices, so "talk to sales" may be the right answer. The rubric scores it as no answer.
What should I ask my own vendor?
Ask for resolution rates on anonymous, pre-login visitors. Ask how the vendor defines an "answered" question. Ask what the bot does when it lacks the content. Then run the three questions yourself.
Button menus and "contact us" handoffs can be a safe default, because the test counted a wrong answer as a failure too. The trade-off is that deflection looks like containment in some dashboards. Ask whether the vendor's resolution metric counts a visitor who clicked a link and left. Also ask who owns the answer content: you, the vendor, or nobody.
Frequently asked questions
Does 4 of 68 mean most chatbots are useless?
No. It means most failed a strict three-question test as an anonymous visitor. The sample, the grading rubric and the first-attempt rule all shape that number.
Is this research peer reviewed?
Nothing in the text we saw suggests it is. It is one author's test, published by CustomerThink, with a small consistency check.
Should I run this test on our bot?
Yes, and run it from a clean browser with no login. It costs an afternoon. If the bot can't state your hours, billing steps and pricing approach, fix the content before you buy more AI.
Sources: CustomerThink, "Only 4 of 68 public company chatbots answered three basic visitor questions" by Luke Wilcox.
This may seem like a basic concept, but the data indicates that it’s worth emphasizing: chatbots should be able to answer the questions your visitors and customers are asking.