All posts

6 min read

AI personas told they had never heard of a product still rated the real one higher than a made-up twin

A Dalpha study ran 145,560 trials across five model endpoints. A Netflix paper the same day found the same pattern in vision models. Here is what that means if you test your store with AI shoppers.

Mantej Gill, author of AI personas told they had never heard of a product still rated the real one higher than a made-up twinMantej Gill

Simulated shoppers are now part of how stores test changes. Shopify’s Sidekick can draft a storefront change and compare it with your current storefront using AI-simulated shoppers. Shopify’s changelog entry of Oct 28 is clear about the limit: “These recommendations are not tested on real shoppers and are not live A/B tests.”

Two papers posted on Oct 29 test a question underneath that. When you tell an AI to play a shopper who knows less than it does, does it actually know less?

I do causal ML research at Carnegie Mellon, and this is the first thing I would check in any simulated shopper.

What the Dalpha team tested

The authors are Sangmin Lee, Sangwon Lee, Sungbeom Choi, Jeeseung Park and Kibeom Myoung, all at Dalpha Inc. They paired real but obscure items with made-up twins that look almost the same. One real pair from the study is Psyllium husk and Psyllyer humb. The analysed set is 178 real items that kept a screened twin.

AI personas rated items on a 1 to 7 scale. In one condition each persona was told, in a sentence, that it had never heard of the item. The design comes to 145,560 trials across five endpoints from two providers.

Asked directly, they said they did not know

The personas did not claim to know the real items. Asked directly at two endpoints, they claimed to know 0 of 96 times at each.

Their ratings told a different story. With the explicit “never heard of it” sentence, real items with a favourable record were rated 0.324 points higher than their twins on the 1 to 7 scale. 72 of the 88 favourable items were rated above their own twin.

Brands were near the top

Split by domain, brands and food additives had the highest point estimates on favourable items, at +0.505. Civic organisations had the lowest, at +0.052. The ranges around these estimates overlap, and the paper does not rank the domains. For brands with an unfavourable record, the real name still came out 0.631 points ahead on a scale that asks about purchase intent.

The authors add a warning about averages. For unfavourable items the sign flips between domains, so a pooled result near zero can hide opposing effects.

Telling is not seeing

A second paper posted the same day, by Jiawei Chen at Netflix, tested personas in vision-language models. Telling a model it had blurry vision did not change what it recognized: “Describing blur leaves recognition unchanged in all 14 models we test.” Blurring the photo itself lowered accuracy in every model, as it does for people.

The limits

The Dalpha authors write, “We never measure a genuinely low-knowledge human,” so they cannot say which way the bias would push a real survey. Real and made-up names differ in spelling, and they say part of the contrast may come from the form of the name rather than knowledge of the item. The five endpoints are commercial APIs from two providers, so the results may not carry over to open-weight models.

Chen tested short template personas. The blur test is a controlled proxy, not clinical low vision, and several of the human reference groups are small.

Neither study had personas shop a store. They rated items or answered questions.

What this means if you test with AI shoppers

My read is that the first-time visitor you simulate may already know the big names in your category. That can tilt a test toward names the model already knows.

  • Watch what the AI shopper does, not only what it says.
  • Put the change on the page it actually sees.
  • Check how it ranks your tests against your past A/B results before you trust it.
  • Let the real A/B test make the decision.

These are our suggestions. Neither paper tested them.

How Shopthru AI helps

Shopthru AI is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. You approve each fix. Autopilot is coming. Then it checks again. Instagram and Amazon are next.

Sources

See what AI would buy from you.