One planted web page got into ChatGPT, Gemini and Perplexity answers 31% of the time

The authors, Michele Guerra and colleagues, call a topic with little or no reliable content a data void. They wrote 90 invented claims, kept 56 for the final study, and published one web page on each. The topics covered business, health, law, politics and technology. The pages were written with ChatGPT-5.2 and checked by two of the authors.

They then asked three web-searching tools about each topic: ChatGPT, Gemini and Perplexity’s Sonar Pro. Each question was asked in three tones, neutral, excited and skeptical. The authors waited until answers were stable, which took 25 days after launch, and collected 504 answers. A page counted as a success if the answer cited it, used it or listed it.

Across all 504 answers, the planted page appeared 31% of the time. In the paper’s words, that is 28.6% cited, 1.0% used and 1.4% listed. Each void was filled with a single page.

The authors say they cannot compare this with topics that already have plenty of content. Their design does not allow a fair comparison. So 31% is not a measure of how easily AI is fooled in general.

The authors hosted pages at three levels of visibility. High was a domain more than six years old, with backlinks and automatic indexing. Medium was a new GitHub Pages domain that the authors submitted to Google Search Console by hand. Low was a new GitHub Pages domain with no submission, public but not pushed to search engines.

Low-visibility pages never succeeded, on any of the three tools. On the old, automatically indexed domain, Gemini used the planted page in 78.3% of its answers. Counting only medium and high visibility, the overall rate rose to 42.3%.

The authors wrote skeptical prompts with phrases like “I have some doubts about…”. Skeptical prompts still pulled the planted page in 30.4% of Gemini answers, 26.8% of ChatGPT answers and 26.8% of Sonar Pro answers. Neutral prompts got 32.1%, 28.6% and 32.1%.

The paper says skeptical prompts slightly reduce success, but it finds no evidence of large effects from tone or model choice. It also says smaller effects cannot be ruled out. Prompts that borrowed words from the planted page did worse for the user: they raised success to as much as 39.3%.

Of the 149 answers that cited or used the planted page, 40 carried a hedge or warning, which is 26.8%. The paper says 73.2% appeared without one. Neutral prompts got the fewest warnings. Even with skeptical prompts, ChatGPT flagged 53.8% and Gemini 29.4%.

The same week, researchers at ZOZO Research in Tokyo posted a different preprint, “Preference Is Not Permission”. They tested sixteen instruction-tuned models on fashion and hotel cases. Fifteen annotators played the shopper and said how far an agent should go without asking. The paper says the models commit even when fit is low and when their own evidence flag reports that product information is missing, and calls this overreach.

This is a benchmark with role-playing annotators, not a field test of live shopping agents. But the lesson sits next to the first paper. When product information is missing, some agents act anyway.

A store is not a data void. Your products have pages, reviews and often competitors. But a small void exists for every question your site never answers: sizing for a tall shopper, an ingredient, how to wash it, how returns work. That is my reading, not a finding of either paper. If your page is silent, an answer can lean on whichever page speaks.

The study’s other lesson is that the fix is not on the shopper’s side. Skeptical prompting helped little. The answer your shopper gets depends on what is published.

Five steps, about an hour for one product.

Shopthru.ai is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. It ships only on your yes, then checks again. Instagram and Amazon are next.