Real car shoppers typed a median of 3 words into a natural-language search. AI models playing shoppers averaged 17.6.
A study with Autotrader UK put one month of real searches next to 118,098 written by AI models. The two looked nothing alike. Here is what that means for anyone testing their store with AI-written questions.
Many teams now test their store with questions an AI writes while pretending to be a shopper. A new paper checks how close those questions come to what real people type. On one live search box, the answer was: not close.
What the study compared
The paper is “Useful Diversity or Hallucinated Variety?” by Filippos Ventirozos and Matthew Shardlow of Manchester Metropolitan University, with Tom Armitage, Mahmoud Oshagh and Peter A. Appleby of Autotrader UK. It was posted to arXiv on Oct 26, 2026 and accepted to the Findings of IJCNLP-AACL 2026.
Its main job is training a search parser for a 47-field car schema. To do that, six AI models wrote 118,098 synthetic searches while playing shoppers. The models were GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, DeepSeek-R1-Distill-Llama-70B, Hunyuan-A13B-Instruct and Qwen3-Next-80B-A3B-Thinking. The authors compared them with 1,214 searches people wrote for the study, and with 15,304 real searches: one month of natural-language queries sent to Autotrader UK’s deployed search.
How long real searches are
Real searches averaged 4.8 words, with a median of 3. The AI-written ones averaged 17.6 words. Even people writing searches for the study averaged 14.3.
47.3% of real searches were two words or shorter. That was true of 0.9% of the AI-written searches and 0.4% of the study’s human-written ones. Real examples in the paper include “electric corsa”, “BMW 330i” and “wheelchair accessible”. 83.0% contained no operator word at all, such as “under” or “within”.
What slipped when AI wrote them
The AI-written searches also changed the meaning of what they were asked to say. The authors give an example: a minimum price of £500 became “under £500”. A descending price sort became “lowest to highest”.
The authors looked closely at the 927 synthetic searches that every parser missed. A committee of AI models labeled 345 of them (37.2%) as hallucinated, meaning the search no longer matched what it was meant to ask. Dropped constraints were the most common sign, at 57.3% of the hallucination cues.
Fair limits
This is one UK car marketplace, not a store, and the authors list domain and UK cultural exposure among their limits. Part of the length gap is by design: the AI was asked to fit more details into each search. On the paper’s 23-field basis, synthetic searches carried 5.10 fields on average and real ones 2.36. The real figure covers the 12,610 searches the live engine could parse.
The 37.2% covers only the hardest cases, the 927 every parser missed. The labels came from an AI committee. After adjudication, the author agreed with the committee on 83 of 100 checked cases. A fully blind first pass agreed on 72 of 100.
What a store can do with this
Our read: the shopper we picture writes sentences, and the one on the site types two or three words. If you test with AI-written questions, check them against real ones first. The authors’ own advice is narrower and worth keeping: check that words like “under” and “within” survive, because they carry the meaning.
The steps below are ours. They are a way to start, and the study does not test them on a store.
How Shopthru AI helps
Shopthru AI is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. You approve each fix. Autopilot is coming. Then it checks again. Instagram and Amazon are next.