Almost no DTC brand blocks ChatGPT. So getting in is not your edge.
We read the robots.txt and llms.txt of 75 well-known DTC brands. Nearly all of them let AI in. One block looks like an accident, and 37 of the 57 Shopify stores serve the same agent guide. Here is what we found.
- Name AI bots and let them in16
- No AI rules, open by default57
- Block training crawlers only1
- Block every AI bot, likely by accident1
See the numbers
| Group | Brands |
|---|---|
| Name AI bots and let them in | 16 |
| No AI rules, open by default (2 have no robots.txt) | 57 |
| Block training crawlers only (Buck Mason) | 1 |
| Block every AI bot, likely by accident | 1 |
A year ago the big AI question for a brand was whether to let the bots in at all. For DTC brands that question is mostly settled. We checked the files that decide it.
What we checked
robots.txt is the file that tells crawlers where they may go. llms.txt is a newer plain text file that tells AI where to look. On Oct 13 and 14, 2026 we read both on 118 well-known US DTC brands, with a research bot that said who it was. We could read the files of 75, and those 75 are the counts below. The full method is at the end.
Almost everyone lets AI in
74 of the 75 brands let OAI-SearchBot, the crawler behind ChatGPT search, reach their product pages. 73 let GPTBot, the crawler OpenAI uses for training. ClaudeBot and PerplexityBot get the same welcome. 2 brands have no robots.txt at all, which means every page is open.
Blocking GPTBot does not take you out of ChatGPT search. OpenAI runs them as separate bots: GPTBot collects content for training, and OAI-SearchBot decides what can be shown in ChatGPT search answers.
One brand says no to training, yes to shopping
Only 2 of the 75 block an AI crawler. Buck Mason is one, and it is a clear choice: its robots.txt blocks GPTBot, ClaudeBot and CCBot, the crawlers that collect content for training, and lets OAI-SearchBot, ChatGPT-User and PerplexityBot in. No to training, yes to search and shopping.
One block looks like an accident
The other brand blocks every AI bot from every page, while letting Google and Bing in. The cause is one line: Disallow: /*#. Under the robots.txt standard, everything after a # is a comment, so a crawler reads the line as Disallow: /*, which means every page. It was most likely meant to block links that contain a #. We are not naming the brand.
17 brands name AI bots, and all 17 let them shop
17 of the 73 robots.txt files name at least one AI bot. Every one of the 17 lets at least one AI bot reach its product pages. 15 give at least one AI bot an explicit Allow rule. The other 2 name a bot only to keep it out of private pages, such as accounts and the cart.
Fenty Beauty goes furthest. Its robots.txt lists 24 bots by name, most of them AI agents, from GPTBot and ClaudeBot to Grok and Amazonbot, under a single rule: Allow.
Most Shopify stores serve the same agent guide
Of the 57 Shopify stores we read, 44 serve an llms.txt. 37 of them serve the same guide, titled Agent Instructions, with only the store name, web address and policy links changed. It tells an AI agent how to shop the store for its user: through the Shop skill, with checkout by Shop Pay, or through the store’s Universal Commerce Protocol endpoint. The guide points to Shopify’s own skill and docs, which suggests Shopify provides it. We have not confirmed that with Shopify.
Off Shopify, 6 of the 18 brands serve an llms.txt. So 37 of the 57 Shopify brands we checked have the same agent guide, whether or not they wrote it.
Getting in is not the edge
If almost every brand lets AI in, being crawlable sets nobody apart. What sets you apart is what AI finds once it is inside: a clear price, stock, a rating it can trust, and shipping and returns written as text. We measured that too, on product pages from the same list of brands, in Your star rating may be invisible to ChatGPT.
And robots.txt is only a request. 11 of the 118 sites refused our labelled research bot at the front door. Our bot is not a shopping agent, and a firewall may treat one differently. But a welcome in robots.txt does not guarantee the bot wall agrees. We covered that in The CAPTCHA that ate Christmas.
Check yours in ten minutes
Open yourstore.com/robots.txt and yourstore.com/llms.txt in a browser. Then tick these off.
How Shopthru.ai helps
Shopthru.ai is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. It ships only on your yes, then checks again. Instagram and Amazon are next.
How we measured
- When
- Oct 13 to 14, 2026
- Brands
- 118 well-known US DTC brands. 75 are counted.
- Not counted
- 25 rate-limited us, 11 refused our bot, 4 did not load, 1 now redirects to another brand’s site, and 2 served a web page instead of a robots.txt.
- Our bot
- Identified itself as ShopthruResearch/1.0, one request per second per site at most, retries spread hours apart.
- robots.txt
- Read with the standard’s rules (groups, wildcards, longest match wins). 12 AI bots checked on the homepage and on a product page path.
- llms.txt
- Counted only when the file is plain text, not a web page.
- Shopify
- Detected from the store’s headers, page code or public products feed.
- Brands named
- Fenty Beauty and Buck Mason, re-checked live on Oct 14, 2026.