AI personas told they had never heard of a product still rated the real one higher than a made-up twin
A Dalpha study ran 145,560 trials across five model endpoints. A Netflix paper the same day found the same pattern in vision models. Here is what that means if you test your store with AI shoppers.
Simulated shoppers are now part of how stores test changes. Shopify’s Sidekick can draft a storefront change and compare it with your current storefront using AI-simulated shoppers. Shopify’s changelog entry of Oct 28 is clear about the limit: “These recommendations are not tested on real shoppers and are not live A/B tests.”
Two papers posted on Oct 29 test a question underneath that. When you tell an AI to play a shopper who knows less than it does, does it actually know less?
I do causal ML research at Carnegie Mellon, and this is the first thing I would check in any simulated shopper.
What the Dalpha team tested
The authors are Sangmin Lee, Sangwon Lee, Sungbeom Choi, Jeeseung Park and Kibeom Myoung, all at Dalpha Inc. They paired real but obscure items with made-up twins that look almost the same. One real pair from the study is Psyllium husk and Psyllyer humb. The analysed set is 178 real items that kept a screened twin.
AI personas rated items on a 1 to 7 scale. In one condition each persona was told, in a sentence, that it had never heard of the item. The design comes to 145,560 trials across five endpoints from two providers.
Asked directly, they said they did not know
The personas did not claim to know the real items. Asked directly at two endpoints, they claimed to know 0 of 96 times at each.
Their ratings told a different story. With the explicit “never heard of it” sentence, real items with a favourable record were rated 0.324 points higher than their twins on the 1 to 7 scale. 72 of the 88 favourable items were rated above their own twin.
Brands were near the top
Split by domain, brands and food additives had the highest point estimates on favourable items, at +0.505. Civic organisations had the lowest, at +0.052. The ranges around these estimates overlap, and the paper does not rank the domains. For brands with an unfavourable record, the real name still came out 0.631 points ahead on a scale that asks about purchase intent.
The authors add a warning about averages. For unfavourable items the sign flips between domains, so a pooled result near zero can hide opposing effects.
Telling is not seeing
A second paper posted the same day, by Jiawei Chen at Netflix, tested personas in vision-language models. Telling a model it had blurry vision did not change what it recognized: “Describing blur leaves recognition unchanged in all 14 models we test.” Blurring the photo itself lowered accuracy in every model, as it does for people.
The limits
The Dalpha authors write, “We never measure a genuinely low-knowledge human,” so they cannot say which way the bias would push a real survey. Real and made-up names differ in spelling, and they say part of the contrast may come from the form of the name rather than knowledge of the item. The five endpoints are commercial APIs from two providers, so the results may not carry over to open-weight models.
Chen tested short template personas. The blur test is a controlled proxy, not clinical low vision, and several of the human reference groups are small.
Neither study had personas shop a store. They rated items or answered questions.
What this means if you test with AI shoppers
My read is that the first-time visitor you simulate may already know the big names in your category. That can tilt a test toward names the model already knows.
- Watch what the AI shopper does, not only what it says.
- Put the change on the page it actually sees.
- Check how it ranks your tests against your past A/B results before you trust it.
- Let the real A/B test make the decision.
These are our suggestions. Neither paper tested them.
How Shopthru AI helps
Shopthru AI is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. You approve each fix. Autopilot is coming. Then it checks again. Instagram and Amazon are next.
Sources
- Lee, Lee, Choi, Park, Myoung (Dalpha Inc.), Can LLMs Ignore What They Know? Measuring Knowledge Leakage in Persona-Based User Simulation, arXiv 2610.39939, Oct 29, 2026
- Jiawei Chen (Netflix), Telling Is Not Seeing: Textual Personas Rarely Make Vision-Language Models Answer Like the People They Describe, arXiv 2610.39331, Oct 29, 2026
- Shopify Changelog, Get AI tested storefront suggestions from Sidekick, Oct 28, 2026