On November 5, Visa published the first results from the Visa Agentic Commerce Benchmark, a test of whether an AI shopping assistant can work out what a shopper needs before recommending what to buy. Across 990 scored runs, 57.2% passed.
Most coverage led with that number. The more useful one sits underneath it. Of the 424 runs that did not pass, 229, or 54%, still ended in a simulated purchase decision that failed Visa's full pass standard. No checkout or payment was attempted and no transaction took place. A purchase here means the benchmark's simulated shopper committed to an offer, and a failed one is not necessarily the wrong product.
How the 990 benchmark runs ended
57.2%of runs passed
A run passed only if the simulated shopper ended with a correct decision and the assistant had uncovered every requirement behind it.
Hover or tap a segment for its count.
- Passed 57.2%
- Purchase decision did not pass 23.1%
- No purchase, though a fit existed 11.5%
- No decision 8.2%
Show the numbers
| Passed | 566 | 57.2% |
|---|---|---|
| Purchase decision did not pass | 229 | 23.1% |
| No purchase, though a fit existed | 114 | 11.5% |
| No decision | 81 | 8.2% |
| Purchase decision did not pass | 229 | 54.0% |
|---|---|---|
| No purchase, though a fit existed | 114 | 26.9% |
| No decision | 81 | 19.1% |
| Purchase decision anyway | 46 | 34.8% |
|---|---|---|
| No purchase | 86 | 65.2% |
| One run passes | 57.2% | |
|---|---|---|
| Two runs both pass | 41.7% | |
| All three pass | 34.2% | |
Agentic commerce has spent this year building controls for the moment money moves: which agent is acting, what the shopper allowed, how much it may spend, which credential it carries, whether the merchant accepts it. Visa's benchmark tests something earlier: whether the agent knows enough to act at all.
What Visa actually tested
The setup, described in Visa Research's whitepaper, is a conversation between two models. The model under test plays the shopping assistant. Another model plays the customer, who holds a private dossier of requirements. Most of those facts start locked. A separate matcher releases one only when the assistant asks the relevant question or requests the right action, such as checking with a spouse.
The assistant searches a controlled market that is isolated from the live web, built from real product listings, prices, seller policies and reviews, and identical for every model. Twenty-two models from 12 labs each ran 15 episodes three times, across TVs, laptops and personal-care appliances. Ten of the 15 episodes are TVs. One, a family replacing a 65-inch set, is public. The other 14 are private.
A run passes only if the customer's final decision is correct and the assistant uncovered every requirement that decision depends on. A selection that happens to match the answer key without those facts fails. Declining to buy passes only when nothing on the market fits.
Each episode ends when the simulated customer selects an offer or declines. Checkout, payment and order placement are outside the current scope.
Newsletter
Follow the research
Research notes and analysis on how AI, digital transformation, product discovery, and customer behavior are changing commerce.
Subscribe to Hu's Weekly HootThe failure that matters
Visa's results page splits the 990 runs four ways. 566 passed. 229 ended in a purchase decision that did not pass. 114 ended with no purchase although a suitable offer existed. 81 ended with no decision, mostly from hitting the tool-call limit or breaking the reply format. 229 of 424 failures is the 54%.
Visa counts a purchase decision as failed in two cases: the offer missed an identified requirement, or it was chosen before every requirement came out, even if it happens to fit. Visa does not publish an aggregate split between products that violated an identified requirement and products selected before every required fact was established.
The sharpest evidence comes from two episodes where nothing on the market meets every requirement, so the correct answer is to buy nothing. By my calculation from the per-model results Visa publishes, not one of Visa's headline findings, 46 of 132 runs on those episodes still ended in a purchase decision, about 35%. Since nothing fit, every one of those selections missed a requirement.
The whitepaper's working hypothesis explains why. Capable models seldom misread a fact once they have it. They fail earlier, by not asking the question that would reveal it, or by treating the shopper's silence as the absence of a constraint. Two of the five failure patterns Visa designs for are treating a value missing from a listing as acceptable and declaring that nothing fits while a suitable offer sits unexamined.
Permission is not knowledge
Each control the industry is shipping answers a different question. Walmart's Agent Profiles let a customer give an agent a defined spending limit or require approval for every purchase. A limit says how much an agent may spend. It says nothing about whether the TV it picked supports the format the family's phone records in. A tokenized credential proves the agent may pay. Merchant acceptance proves the merchant will take the order. Every layer can pass while the decision behind it is incomplete or wrong.
This is the gap Anthropic's Project Swap exposed in September, when 85% of a simulated market's shortfall came from imperfect models of what people wanted rather than from bargaining. Visa measures the same gap from another angle: not whether the agent ranks preferences correctly, but whether it asked enough to rank them at all.
It also explains why the Visa, BILL and Billtrust pilot is a sensible place to start. There, the agent executes an invoice payment the buyer already approved, and every requirement is written down before it arrives. A consumer shopping agent has to discover unstated needs, compare offers, decide whether anything fits and know when to stop. When NPCI's chairman argued that AI may recommend but should not decide that money moves, the recommendation was the part allowed to be uncertain. This benchmark shows how uncertain it is.
What the numbers do not show
Visa designed, ran and scored this benchmark, and it sells infrastructure for agentic commerce. Nobody has reproduced the results, and 14 of the 15 episodes cannot be inspected. The categories are narrow and weighted toward TVs. The shopper is a model, not a person; Visa says it plans to calibrate it against recorded human sessions. The pass standard is deliberately strict, so a run fails when the agent skipped a task-defined requirement even if the product fits, which means the 42.8% failure rate is not a count of wrong products or of purchases that would harm a consumer. The confidence intervals of the next 11 models overlap those of the top two. The five costliest models passed 67.1% of runs and the five cheapest 50.2%, which shows correlation, not that price causes better shopping.
Across models, all three runs of the same episode passed only 34.2% of the time. A demonstration that works once proves little.
What operators should take from it
None of this argues against agents buying. It argues for a control that sits before payment authorization: a stop rule. Visa's own Trust Index found that 23% of US consumers trust AI to handle payments for them. The benchmark makes that trust problem more concrete: authorization does not help if the decision underneath it is incomplete. These are implications from the evidence, not controls Visa has tested:
- Confirm hard constraints explicitly, and separate required attributes from preferences.
- Treat a missing product attribute as unknown, not as a match.
- Keep buying nothing as a valid, logged outcome.
- Require fresh approval when the product, price or terms change.
- Route expensive, ambiguous or hard-to-return purchases to a person.
- Record which requirements were established before the agent acted.
- Test the same task repeatedly, not once.
Visa says later phases will extend the benchmark to checkout, payment and order placement. Those layers have more mature controls, but Visa has not tested them in this benchmark. The missing one is the layer that says not yet.
If your agent could buy only when it could show what it had established, how often would it be allowed to act?
