In Visa’s AI shopping benchmark, about half of the failed runs still ended with a purchase
Visa tested 22 AI models on 15 shopping challenges. Models failed 42.8% of runs, and about 54% of those failures still bought something. Here is what it measured and what it did not.
On Nov 5, 2026, Visa released the Visa Agentic Commerce Benchmark. It tests AI models on realistic buying tasks. To pass, a model has to find out what the shopper needs through conversation, then buy the right product, or decline to buy if nothing fits.
A disclosure first. Visa is a payments company and sells agentic commerce products under the name Visa Intelligent Commerce. Visa says the benchmark is not part of that portfolio and calls it complementary research. Every number below is from Visa’s release and its Trust Index page.
How the benchmark works
The first round put 22 AI models from 12 labs through 15 shopping challenges, such as buying a TV, a laptop or a personal care appliance. Each model got three attempts at each scenario.
Visa gives an example. A shopper wants help choosing a TV. They mention a relative’s brand recommendation and a membership, and leave other details out. The model has to uncover the budget and the rest, such as format support, HDR10+ and gaming.
Almost half of the runs failed
The overall pass rate was 57.2%, so models failed 42.8% of runs. The best model passed 82.2% of attempts. The worst passed 28.9%.
Passing once did not mean passing again. Across all models and scenarios, all three attempts passed only 34.2% of the time.
Failing, and buying anyway
This is the number that matters most for a store. About 54% of the failed runs still ended with the AI making a purchase. Either it bought something that did not match the shopper’s needs, or it bought before it had found them all.
Asking helped. The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate. The five that uncovered the least passed 44.4%. Cost helped too: the five most expensive models per run passed 67.1% of runs, against 50.2% for the five cheapest.
What it did not test
This round covers finding out what the shopper needs and choosing to buy or not. Visa says it will extend testing to checkout, payment and order placement, and add more scenarios and categories. Visa also says it is not claiming AI agents are not ready for commerce. It names reliability and consistency as the main challenges.
Shoppers are cautious too. Visa’s Trust Index, a Harris Poll survey of 2,065 U.S. adults in May 2026, found that 72% had used an AI assistant. Only 23% trusted AI to handle payment transactions on their behalf.
What it means for a store
Our read, not Visa’s: a wrong purchase made by an agent is still an order from your store. The return, the support ticket and the bad review land with you, not with the model. The facts an agent needs to choose correctly, such as specs, compatibility and what is in the box, should be easy to find and correct on your pages.
How Shopthru AI helps
Shopthru AI is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. You approve each fix. Autopilot is coming. Then it checks again. Instagram and Amazon are next.