# ShopBench — does a price reference make a shopping agent more rational?

Run `run5` · model `openai/gpt-6-luna` · 60 tasks × 4 conditions · 2026-10-09

| condition | optimal choice ↑ | valid choice ↑ | grounded ↑ | median overspend ↓ | price error ↓ | tool calls | time | cost/task |
|---|---|---|---|---|---|---|---|---|
| No tools (memory) | 12% [6–22] | 22% [13–34] | 23% [14–35] | -20% | 66% | 0.0 | 5s | 0.0¢ |
| Web search + fetch | 22% [13–34] | 33% [23–46] | 37% [26–49] | -32% | 25% | 3.7 | 23s | 0.2¢ |
| Prices API | 60% [47–71] | 82% [70–89] | 83% [72–91] | 0% | 0% | 1.2 | 8s | 0.1¢ |
| Web + Prices API | 68% [56–79] | 75% [63–84] | 82% [70–89] | 0% | 0% | 1.5 | 11s | 0.1¢ |

Rates are % of tasks with 95% Wilson intervals in brackets. Definitions, dataset and per-task results: index.html.

![Figure 1](outcomes.png)
![Figure 2](overspend.png)
![Figure 3](cost.png)