In 97,439 test shopping trips, an AI agent failed at reading product facts 37.5% of the time. Paying failed under 1%.
General Machines released CartTrace-100k on Oct 28. It is a sandbox with one model, and the authors say so. Here is what it measured, what it cannot say, and what a store can do with it.
Most of the agent-commerce preparation we hear about is about checkout. A new dataset from General Machines looked at the whole journey, from the shopper’s request to a paid order, and found the losses in an earlier place. The authors are Pratham Bhatnagar and Deepanshu Pal. The release is dated Oct 28, 2026.
A disclosure before the numbers. General Machines says on the same page that Machine Commerce is its agentic commerce arm. They are a company working in this space, and this is their own study. We read both the summary page and the full paper (PDF) and took every figure below from them.
What CartTrace-100k is
It is 101,416 journeys by one open-weight agent, Qwen3.8-27B, across 1,044 stand-in stores in 25 product categories. There are five protocol surfaces and five payment rails. Each journey runs against a simulated shopper, a program that reveals some needs only when the agent asks for them.
Everything runs in a sandbox. The authors say real store data cannot be published without each store’s permission, and nothing may be bought on a real store. Each stand-in store borrows only a real store’s category, market and way of selling. Products come from open catalogs, and prices and stock are synthetic. Payments use test tokens.
Where journeys were lost
Over all 101,416 journeys the agent succeeded 68.7% of the time. A scripted reference agent that knows each task reached 82.2%. The authors then focus on the 97,439 full-data journeys, where the task can be done and the shopper is built to pay.
On those, two of the judge’s ten stages carry the loss. Reading product facts failed in 37.5% of journeys. Clarifying the shopper’s needs failed in 28.9%. Building the cart, checking out and paying each failed in under one percent. Once the shopper approved a charge, 12 of 41,969 approved charges were not completed.
Carting without opening the product page
The authors say a read-facts failure is almost always a journey in which the agent carted without opening a product page. That habit showed up in 36.2% of successful journeys and 40.6% of failed ones, so it is common either way and a little more common when things went wrong.
It also showed up in refusals. The agent refused 4,419 of 7,655 satisfiable tasks without opening a product page. Across platforms, the share of journeys that skipped the product page ran from 20.5% on Medusa to 46.8% on Shopify. The authors do not claim that this ranks the platforms, and the cells differed in their mix of tasks.
Giving up and asking
The agent gave up on tasks the store could serve. In 7.9% of satisfiable journeys it ended by refusing. When the named product was out of stock and a similar one would do, that rate was 27.4%. The authors add a caution: in a first, model-read sample of 30 refusals, 19 traced to a fault in the task or the judge.
Asking questions helped only where a need was hidden. With two needs kept back, journeys where the agent asked succeeded 62.0% of the time, against 21.2% where it did not. The authors read this as a matter of targeting, not frequency, and note that these are associations within a task, not the effect of forcing a question.
The store explained little of the result
This is the caveat that matters most for a store owner. The authors report that the design factors explain 7.5% of the variance in success, the stand-in store 1.9% and the task within its store 52.4%. One run of a task is not a measurement either: of 24,552 tasks run at least three times, 29.3% both succeeded and failed. With about 93 journeys per store, no store is scored, named or ranked.
Their own limits section is direct. One agent ran, so the levels are Qwen3.8-27B’s. The sandbox removes what makes live checkout hard, such as bot walls, logins and pages that change. The shopper is a program built to pay, and they cite work saying LLM-simulated customers understate walking away. They call the near-zero payment failure a property of protocol calls in a sandbox, not a claim about the live web.
What a store can do with this
The authors call their suggestions observations of one agent on stand-ins, not a tested recipe. They say to measure where agent journeys stop, put the deciding facts (size, color, stock, compatibility, delivery) in the search result where the agent reads first, and, when a named item is out of stock, offer in-stock alternatives and say what they share with it. Then measure again with the same tasks.
The steps below follow that advice. They are a way to get started, and the study does not show they raise sales.
How Shopthru.ai helps
Shopthru.ai is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. You approve each fix. Autopilot is coming. Then it checks again. Instagram and Amazon are next.