0 of 122 Shopify stores passed an AI shopping agent test
Checkout worked at every store that was tried. But asked what shipping to Boston costs, none of 88 stores gave a usable answer, and 51 block their own policy pages from bots.
On Oct 14, Search Engine Journal published an audit by Bill Hunt, president of Bisan Digital, titled “Your Store Is ‘Agent-Ready’ By Default, We Tested 122 Shopify Stores To See What That Actually Means.” Two days later, BrightEdge, an AI search software company, published research on which pages ChatGPT cites. Together they point at the same gap.
What Hunt tested
Hunt audited 122 leading Shopify storefronts, picked from published rankings of top Shopify merchants, between August and September 2026. His question was not whether AI can find or cite these stores. It was whether an agent can resolve a purchase decision on them: enough information, context, rules and ability to act to answer the buyer’s questions and commit.
He ran nine checks. Zero of 122 stores passed all nine. The best three reached seven of nine, and all three failed the same two: no stated cancellation window, and no way to cancel.
Checkout worked every time
Credit first. Hunt ran a live checkout test that never completed a purchase or sent payment details, on every store that exposed Shopify’s Universal Commerce Protocol endpoint. It created a checkout for one in-stock item going to a Boston address, read it back, cancelled it, then cancelled it again.
It worked at all 74 stores tested: checkout created, status returned at each step, errors coded, cancel confirmed, and a repeat cancel refused with a coded error. A handoff link came back so the agent could pass to a human. Hunt gives Shopify credit for deploying an agent-callable commerce layer at this scale.
One catch. All 74 checkouts came back as “requires escalation.” On Shopify today, the agent builds a priced cart and a person commits the money. That matches Shopify’s default rule that checkout needs human approval.
The questions that decide the sale
Hunt then asked each store’s own policy tool questions a buyer would ask. “Returned anything” means the tool replied. “Usable” is Hunt’s judgment of whether the reply answered the question.
| Question to the policy tool | Returned anything | Usable answer |
|---|---|---|
| What is the return window for this item? | 88 of 88 | 70 of 88 |
| Can used or open-box items be returned? | 88 of 88 | 5 of 88 |
| Can I cancel an order after placing it, and by when? | 8 of 88 | 2 of 88 |
| What does shipping cost to Boston, and when will it arrive? | 1 of 88 | 0 of 88 |
The return window is the one clear success. For the rest, Hunt says the content is often there, but the tool cannot reach it when the question sounds like a buyer rather than a policy page. One store’s FAQ says “free standard ground shipping on U.S. orders over $75,” and that answer never came back for a question about what shipping costs.
Two stores answered a cancellation question with their shipping FAQ. Hunt’s warning: a wrong answer that looks confident is more dangerous than no answer. And no store returned an arrival estimate or a carrier, across all 74 live checkouts. Shipping cost to Boston was priced in a live checkout at 36 of 74 stores.
51 stores block their own policy pages
Hunt found that 51 of the 122 stores block /policies/ in robots.txt. He notes this is not Shopify’s current default. Shopify’s documentation says the rule persists only on themes with an old robots.txt.liquid template, and that deleting the template restores the default.
Policy pages are where the answers about returns and shipping live. A page that bots cannot open cannot be quoted.
Why this matters in ChatGPT
BrightEdge’s research, published Oct 16, compared the pages that ChatGPT and Google AI Overviews cite for holiday questions. It used the top 50 cited rows per retailer across four major US retailers, for June to August 2026, pulled Oct 14.
Help and policy pages made up 10.8% of ChatGPT’s citation weight, against 0.9% on Google AI Overviews, a ratio of 12.5 times. That held at all four retailers. BrightEdge also found that half of ChatGPT’s top holiday prompts are logistics questions, such as whether a store is open on Thanksgiving.
Read this with care. BrightEdge measures which pages are cited, not what the answers say, and not clicks. Its four retailers are not Hunt’s Shopify sample, so the two studies are not one experiment. The link is ours: shoppers ask logistics questions in ChatGPT, and those answers sit on help and policy pages.
Who owns the missing answers
Hunt sorted the failures by the team that owns the missing information. None of those teams owns the website.
| What is missing | Who owns it, per Hunt |
|---|---|
| Pickup availability and cutoff | Fulfillment and stores |
| Delivery estimate | Fulfillment |
| Cancellation window and ability | Order management |
| Used and open-box exceptions | Legal and customer service |
| Return window as data | Legal |
His conclusion: the checks nearly every store passes are the ones the platform generates. The ones nearly every store fails need someone to write a commercial rule and a system to enforce it.
What to do this week
Most of this takes an afternoon. These steps follow Hunt’s own advice for merchants.
One limit on the audit. Hunt tested shipping to a single Boston address, and he told each store exactly which item to buy. A real shopper rarely arrives with an item ID.
How Shopthru.ai helps
Shopthru.ai is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. It ships only on your yes, then checks again. Instagram and Amazon are next.