All posts

5 min read

Of the AI coding runs that finished the requested page change, 28.2% still broke something else on the site

ProductWebBench graded 13 coding models on two things: did they build the change, and does the rest of the website still work? Here is what it found, its limits, and what it means for a store.

Vishal Verma, author of Of the AI coding runs that finished the requested page change, 28.2% still broke something else on the siteVishal Verma

Coding agents are usually graded on one question: did they build what was asked? A new benchmark adds a second one. After the change, does the rest of the website still work? The authors call this website continuity.

The paper is “ProductWebBench: Evaluating Website Continuity Beyond Task Completion” by Jiajun Wu and 11 co-authors. It was posted to arXiv on Oct 22, 2026. It is the authors’ own research, not a Shopthru AI result. We read the abstract and the full paper and took every figure below from them.

What ProductWebBench tests

It has 320 Change tasks, which ask an agent to modify a runnable website while keeping what already works, and 80 Build tasks, which grow a site over several milestones. The repositories are public and span React, Vue, Svelte, Astro, Gatsby, Hugo, PHP and static sites. Each one must install, build and serve before it is used.

A simulated user reveals the work one stage at a time. Checks that pass at one stage stay in force for the rest of the run. In all, the authors report 8,737 valid runs from 13 models.

Done is not the same as working

Of 674 Change runs that passed every check on the requested work, 190 (28.2%) still failed at least one continuity check. The paper says every model shows it.

The continuity checks look for things like horizontal overflow, console errors, forbidden text and a route that stops rendering. In plain words: the new feature is there, and something else on the site broke.

Breaking what already passed

Some runs went backwards. In 73 of the 674 completing runs (10.8%), a check that had already passed later failed. On the 586 runs where both checks were possible, 107 (18.3%) were caught breaking a verified state. That count joins two checks: 66 runs failed against an untouched copy of the repository, and 57 broke a state they had verified themselves. 16 runs were caught by both.

Looking right is not the same as working, either. Among 1,322 runs whose page structure already met the target, 29.5% failed the interaction checks, the tests that click and type through the page.

The limits

The authors are careful about their own numbers. An audit of the automated grader found a 19.0% false-fail rate, so results after completion depend on the grader’s verdict. Comparisons between models are descriptive, and model coverage is uneven because of provider availability. The untouched-baseline comparison covers 234 of the 320 Change tasks.

These are general websites, not online stores. The paper does not test a cart or a checkout, and it does not measure sales. What follows is our read for a store, not a finding of the paper.

What a store can do with this

If an AI tool edits your theme or a product page, the change you asked for is only half the test. The cart, the product page and the checkout can break while the requested edit looks done. The authors’ own conclusion is that a patch should be scored against the living system it changes, not just its target.

How Shopthru AI helps

Shopthru AI is an AI agents team for consumer brands. It asks ChatGPT, Gemini, Google AI, Perplexity and Copilot what your shoppers ask, finds where AI gets your products, prices or policies wrong, and drafts each fix into an approval inbox. You approve each fix. Autopilot is coming. Then it checks again. Instagram and Amazon are next.

Sources

See what AI would buy from you.