arthgupta.dev · prices · summary.json · tasks.json · results.jsonl · report.md
Summary. On 60 “find the cheapest X that meets spec Y” tasks, the same openai/gpt-6-luna agent chose the optimal product 12% of the time with no tools and 68% with the Web + Prices API (+57 points). Grounded answers went from 23% to 82%; valid purchases (spec met) from 22% to 75%. The benchmark holds the model, prompt and step budget fixed and varies only the information the agent can reach:
nothing, the open web, this site's price API, or both. The reference optimum for each task is the cheapest product meeting the task's minimum spec
that prices.arthgupta.dev has observed across retailers.
| condition | optimal choice ↑ | valid choice ↑ | grounded ↑ | median overspend ↓ | price error ↓ | tool calls | time | cost / task |
|---|---|---|---|---|---|---|---|---|
| Web + Prices API web+prices | 68% [56–79] | 75% [63–84] | 82% [70–89] | 0% | 0% | 1.5 | 11s | 0.1¢ |
| Prices API prices | 60% [47–71] | 82% [70–89] | 83% [72–91] | 0% | 0% | 1.2 | 8s | 0.1¢ |
| Web search + fetch web | 22% [13–34] | 33% [23–46] | 37% [26–49] | -32% | 25% | 3.7 | 23s | 0.2¢ |
| No tools (memory) none | 12% [6–22] | 22% [13–34] | 23% [14–35] | -20% | 66% | 0.0 | 5s | 0.0¢ |
Rates are the share of the 60 tasks; brackets are 95% Wilson confidence intervals. Overspend is the median over valid choices only; price error is the median over grounded answers. Primary metric: optimal choice rate.
Task — You are shopping online in the US. Find the cheapest anklets that meets this requirement: At least one complete anklet intended to be wearable by an adult, with a stated length or adjustable fit. Material and fit should be clear enough for the buyer to judge suitability. Compare products by price per anklet. Answer with the specific product (brand, product name, size), the retailer, the listed price in USD, and the price per anklet.
Reference optimum — Pura Vida Stretch Stage Lights (multi): $10.0000 per anklet ($10.00 at puravida); 7 of 128 products in the reference set meet the spec.
| condition | outcome | agent's answer |
|---|---|---|
| No tools (memory) | ungrounded | PAVOI 14K Gold-Plated Satellite Chain Anklet, adjustable 9 in + 2 in extender — quoted $13.95 at Amazon |
| Web search + fetch | valid, +0% overspend | |
| Prices API | optimal | Pura Vida Stretch Stage Lights Anklet (1 anklet, adjustable fit) — quoted $10.0 at Pura Vida |
| Web + Prices API | spec not met | Pura Vida Stage Lights Stretch Anklet, 9-inch (one size fits most) — quoted $10.0 at Pura Vida |
| metric | definition |
|---|---|
| Optimal choice rate ↑ | the agent's pick is the cheapest product in the reference set that meets the minimum spec (primary metric) |
| Valid choice rate ↑ | the pick is grounded and meets the minimum spec: a correct purchase, cheapest or not |
| Grounded rate ↑ | the named product exists in the reference set; the complement (ungrounded) is a product the reference does not contain: real-but-uncrawled or hallucinated |
| Median overspend ↓ | over valid choices: unit price of the pick ÷ unit price of the optimum − 1 (median; the mean is in summary.json) |
| Price error (median) ↓ | over grounded answers: |quoted price − observed price| ÷ observed price |
Each task has a fixed reference set: every product in that category in the index, with the cheapest spec-satisfying one marked as the optimum. An LLM judge maps the agent's free-text product to a reference item (same product line and size family) or to none; everything else is arithmetic on observed prices. External verification: an answer the index cannot ground is looked up live at the retailers (Amazon search, Walmart catalog), fetched, normalized and graded against the same optimum; if it is a real spec-meeting product it counts as grounded and valid, and if it is cheaper than the index's optimum it counts as optimal and is flagged beats index (a coverage gap the benchmark then fills). Externally grounded answers this run: No tools (memory) 18% (beats index: 6) · Web search + fetch 27% (beats index: 12) · Prices API 0% (beats index: 0) · Web + Prices API 2% (beats index: 1).
60 tasks, one per category with at least four priced products meeting its minimum spec and at least two distinct unit prices. The minimum spec and comparison unit come from the category's schema on the prices site. Every prompt has the same form:
You are shopping online in the US. Find the cheapest fragrance oil that meets this requirement: For home scenting, choose a fragrance oil explicitly labeled for at least one intended home-fragrance use. Diffusers, burners, sprays, potpourri, and candle making may require different formulations, so application labels are not interchangeable. Compare products by price per ml. Answer with the specific product (brand, product name, size), the retailer, the listed price in USD, and the price per ml.
| category | minimum spec | unit | meet spec / items | reference optimum |
|---|---|---|---|---|
| Fragrance Oil | For home scenting, choose a fragrance oil explicitly labeled for at least one intended home-fragrance use. Diffusers, burners, sprays, potpourri, and candle making may require different formulations, so application labels are not interchangeable. | ml | 12 / 24 | Junaid Jamshed Diffuser Oil Midnight Smoke · $104.4000/ml |
| Drinkware | For a basic everyday serving vessel, choose drinkware with at least 250 ml capacity. A lid, handle, insulation, or particular material is not essential for the basic job. | ml | 158 / 239 | Daiso Cold Water Pitcher Green · $2.2690/ml |
| Washing Machines | For most households, look for a fully automatic machine with at least 7 kg of wash capacity. Either front-load or top-load can meet the basic need. | each | 41 / 123 | Godrej Fully Automatic Top Load Washing Machine · $18500.0000/each |
| Hoodies | A basic hoodie for most people has a hood and long sleeves, with a size available for the intended wearer. Fabric, closure, and special features are preferences rather than universal requirements. | each | 40 / 40 | Star Tribune No Regrets Hoodie · $40.9500/each |
| Eyeliners | For a basic everyday eyeliner, look for at least one product in a black or brown shade. Format, finish, and wear claims are preferences rather than requirements. | item | 17 / 132 | Junaid Jamshed Ultra Black Eyeliner · $5.4000/item |
| School Uniform Pants & Shorts | A school-appropriate pair of pants or shorts in the student’s correct size, with leg coverage that meets the school’s dress code. No single material or fit is essential for most students. | each | 41 / 42 | Everlane The Skinny 4-Way Stretch Organic Jean D · $20.0000/each |
| Ventilation Grilles | For most people, a grille needs to match the opening’s shape and dimensions and be suitable for its installation location. A screen or adjustable louvers are optional. | each | 4 / 6 | Air Vent Multi-Pitch FilterVent Roof Ridge Vent · $2.7900/each |
| Sticky Notes | For a typical writable reminder note, choose sticky sheets at least 50 × 50 mm. Paper or film is suitable; smaller index tabs are better suited to marking pages than writing notes. | sheet | 6 / 27 | Daiso Sticky Notes Pastel yellow · $0.0048/sheet |
| Tablets | For general browsing, streaming, and everyday apps, choose a general-purpose tablet with a screen of at least 8 inches and at least 32 GB of storage. E-readers are not substitutes for this minimum. | tablet | 10 / 32 | Amazon Fire HD Black · $154.9900/tablet |
| Joggers | For most adults, a wearable-size pair of full-length joggers is the basic choice for lounging. Fabric, fit, and rise can be chosen to suit personal preference. | each | 24 / 35 | Everlane The Track Jogger Faded Blue · $23.0000/each |
| Bomber Jackets | For everyday use, a bomber jacket should be sized for a baby or child and have long sleeves. Insulation or weather protection is useful but not essential for a lightweight or mild-weather jacket. | each | 7 / 71 | Kith Novelty Souvenir Jacket Stadium · $125.0000/each |
| Cotton Swabs | For common personal-care and household uses, choose swabs with cotton tips on both ends. A standard shape and ordinary stick material are sufficient unless a specific task or preference calls for more. | swab | 20 / 34 | Amazon Essentials Cotton Swabs · $0.0050/swab |
| Sport Jackets | For most adults, a sport jacket should be offered in a size that fits and have long sleeves for everyday outerwear use. Weather protection or insulation is useful for particular conditions but is not essential for every sport-jacket style. | each | 76 / 127 | Everlane Barn Jacket Black · $53.0000/each |
| Cycling Shorts & Bib Shorts | For most riders, the minimum useful option is a correctly sized pair of cycling shorts or bib shorts with a padded chamois. Thermal fabric, bib construction, and extra coverage are preferences rather than requirements for basic riding. | each | 8 / 64 | Red Bull BORA hansgrohe SPECIALIZED Training Bib · $144.9500/each |
| Laundry Pods | A laundry-detergent pod that is compatible with HE washers and provides one dose for a small/medium load. Use the listed dose to compare the price per load. | load | 13 / 45 | Amazon Basics Laundry Detergent Pacs · $0.1798/load |
| Precision Applicator Bottles | For most small-volume dispensing tasks, choose a refillable bottle with a controlled applicator tip and a closure. The tip type should suit the liquid and level of precision required. | ml | 4 / 9 | DAISO スポイトボトル · $14.5200/ml |
| Deodorant | For basic odor control, choose a deodorant or antiperspirant deodorant in a form you are comfortable applying. A stated protection duration is useful for comparison but is not required for the product to qualify. | g | 69 / 117 | Right Guard Xtreme Defense 5 Invisible Solid Ant · $3.1058/g |
| Quilts and Comforters | A basic quilt, coverlet, or comforter in a size that fits the intended adult bed is sufficient for most people. No particular fill, fabric, warmth level, or decorative construction is essential; choose those to suit personal needs. | sq ft | 109 / 178 | Parachute Hand Stitched Cotton Quilt Ash · $1.7222/sq ft |
| Coin Cell Batteries | For most devices designed for a primary CR coin cell, choose a non-rechargeable 3 V lithium-manganese-dioxide battery. Its form-factor code must match the device exactly; other devices may require a different chemistry or voltage. | battery | 5 / 9 | Mitsubishi Lithium Coin Battery · $0.7260/battery |
| Sunscreen | For most people, choose a labeled broad-spectrum sunscreen with SPF 30 or higher. Water resistance is useful for swimming or sweating but is not essential for everyday use. | ml | 31 / 169 | Hawaiian Tropic Sheer Touch Sunscreen Lotion Tro · $33.8140/ml |
| Packaged Drinking Juice | For a general-purpose juice purchase, require ready-to-drink product that is 100% fruit juice. Flavor and pulp preference can vary by buyer. | ml | 7 / 46 | Apple & Eve 100% Juice Variety Pack Variety pack · $2.0860/ml |
| Underwear | A wearable underwear garment of the required type, with a listed size that fits the shopper. For active use, choose a garment with support appropriate to the activity. | garment | 215 / 250 | Unknown Boxer Briefs · $0.7000/garment |
| Powered hand fans and misters | A complete, handheld fan with a portable electrical power source is the minimum for personal airflow. Misting, extended runtime, and multiple speeds are useful upgrades rather than requirements. | fan | 10 / 22 | Daiso USB Fan with Brushless Motor Light Blue · $2.1780/fan |
| Salwar Kameez & Suit Sets | A ready-to-wear set in the wearer's size, with a top and a matching bottom. A dupatta or decorative detailing is not required for a basic complete outfit. | each | 1670 / 1917 | Junaid Jamshed Kameez Shalwar Orange · $2.4300/each |
| Bread | At least an 8 oz package of ready-to-eat bread in loaf, roll, bun, or flatbread form. No specialty nutrition or flavor claim is required for ordinary bread use. | g | 8 / 39 | Amazon Grocery Round Top Bread White · $2.6810/g |
| Coffee Makers | A functioning coffee maker that brews coffee in a recognized format. Capacity and convenience features can vary with serving needs. | brewer | 7 / 12 | Daiso Coffee Dripper · $3.6300/brewer |
| Shoe Charms | For most people, a shoe charm should include at least one decorative piece with an attachment method that fits the intended footwear. Check the specific shoe type before buying, since clog-hole plugs, clips, cords, and chains are not interchangeable. | charm | 15 / 29 | Daiso Shoe Tag Unicorn and flowers · $0.3630/charm |
| Unstitched Suit Sets | For a basic complete outfit, the set should include unstitched material for both a shirt and trousers, with enough yardage for the intended wearer. A dupatta is optional. | each | 31 / 313 | Saya Khaddar Suit Ivory · $4.5000/each |
| Gift Wrap | For most people, a basic gift-wrap product should provide at least one usable piece in a form that can wrap or contain a gift. Check the listed dimensions against the gift; no single size fits every present. | count | 118 / 136 | Daiso Clear Gift Bag Simple design with ribbon · $0.0605/count |
| Dryer Sheets | A dryer sheet that reduces static and softens fabric is the basic fit for most people. Scent, material claims, and single-use versus reusable format are preferences rather than universal requirements. | sheet | 28 / 33 | Snuggle Fabric Softener Dryer Sheets Blue Sparkl · $0.0327/sheet |
| Ashtrays | For most people, a basic reusable tabletop ashtray is sufficient for collecting ash during everyday use. More specialized types are needed for vehicles, outdoor locations, or fixed installation. | ashtray | 17 / 25 | Edie Parker Diner Ashtray Early Bird Green · $8.0000/ashtray |
| Razors | For most people seeking a straightforward manual shave, choose a disposable or cartridge razor with at least three blades. A handle is needed for cartridge refills but not for disposable razors. | razor | 24 / 132 | BIC Soleil 3 Colors · $0.9879/razor |
| Anklets | At least one complete anklet intended to be wearable by an adult, with a stated length or adjustable fit. Material and fit should be clear enough for the buyer to judge suitability. | anklet | 7 / 128 | Pura Vida Stretch Stage Lights (multi) · $10.0000/anklet |
| Pet Furniture | There is no single size or capacity threshold that suits every kind of pet furnishing. For most people, the minimum is a complete product of the appropriate type for the intended pet and use; confirm dimensions and any stated load rating fit the animal or equipment. | item | 25 / 120 | Pet Mat Fluffy boa · $3.6300/item |
| Cat Litter | A product labeled as cat litter, made from a standard absorbent litter material. Either clumping or non-clumping litter can do the job for most households. | g | 4 / 24 | Purina Tidy Cats 24/7 Performance · $0.0019/g |
| Blouses | A blouse available in the wearer's size, with coverage and a fabric suited to the intended setting and climate. No particular neckline, sleeve style, or care method is essential for most people. | each | 6 / 6 | Everlane Linen Billow Blouse Leek · $36.0000/each |
| Phone Chargers | A standard USB charger rated for at least 10 W is enough for basic charging for most people. The output connector must match the phone or its cable; a cable need not be included. | each | 17 / 110 | boAt WCD QC3.0 Fast Charger Adaptor White · $4.1531/each |
| Apparel Outerwear | For a broadly useful everyday outer layer, choose a full-sleeve jacket, coat, parka, bomber, or sports jacket/coat. Insulation, a hood, and weatherproofing depend on climate and intended use, so they are not required for every shopper. | piece | 1171 / 1761 | Unknown UPF50 Sun Protection Jacket · $11.2000/piece |
| Phone Screen Protectors | The protector must be explicitly compatible with the buyer's exact phone model and cover the screen. A basic screen protector is sufficient; specialty filters and edge-to-edge coverage are optional. | each | 7 / 12 | Daiso Glass Screen Protector blue-light filter · $0.7260/each |
| Rompers | A wearable one-piece romper with short legs, in a size and lifestage appropriate for the intended wearer. Sleeve coverage and material can vary by climate and preference. | garment | 20 / 121 | Halara Floral Yarn Cowl-back V-neck Cap Sleeve H · $34.9500/garment |
| Video Games | Choose a working copy of the full game for the buyer's platform. For most people, a standard edition is enough; match the listed platform to the system and check age rating and online requirements for the intended player. | game | 295 / 451 | Jewel Quest Solitaire Trio · $2.3300/game |
| Electronic Circuit Boards | For most buyers, the board should be specified as compatible with the intended device or project and include the components needed for the stated job, either assembled or supplied as a parts kit. A bare PCB is suitable only when the buyer intends to source and install components separately. | each | 42 / 68 | Adafruit DVI Breakout Board · $1.9500/each |
| Bar Soap | A bar intended to cleanse the hands or body is sufficient for routine washing. No particular scent, bar base, or specialty claim is required. | g | 34 / 85 | Irish Spring Bar Soap Icy Blast · $6.3290/g |
| Razor Blades & Refills | For most shoppers, the minimum is at least one replacement blade unit in the correct format for their existing razor handle. Compare prices only within compatible systems. | each | 7 / 10 | Dollar Shave Club Classic 4 Blade · $1.6719/each |
| Watches | For a basic everyday watch, require a conventional timekeeping movement and at least splash-resistant water protection. This is a baseline, not a recommendation for swimming or diving. | watch | 54 / 468 | Baby-G BG169R-2C Blue / Orange · $89.0000/watch |
| Tumblers | A reusable tumbler should hold at least 350 ml, enough for a typical single serving of a cold or hot drink. Insulation, a lid, and a handle are useful preferences but not necessary for the basic job. | ml | 10 / 11 | Stanley Quencher H2.0 Flowstate Purple Dust Spoo · $38.0410/ml |
| Paper Towels | For routine household cleanup, a usable paper towel should have at least one ply, at least one sheet, and perforations for tearing off sheets. Full, smaller, or selectable sheet sizes can all do the job. | sheet | 16 / 53 | Genuine Joe Kitchen Paper Towels White · $0.0065/sheet |
| Bolero Jackets | A wearable cropped outer layer in a size that fits the intended wearer, with sleeves for ordinary bolero-style shoulder and arm coverage. | each | 4 / 4 | Frankies Bikinis x Paige Lorenze Love Story Lace · $120.0000/each |
| Food Jars | For most people carrying a single meal, choose a portable jar with at least 12 fl oz capacity, insulation, and a leakproof closure. | ml | 6 / 34 | Stanley Adventure To-Go Food Jar + Spork Frost · $40.3890/ml |
| Crop Tops | A wearable upper-body garment with a cropped length, available in a size that fits the shopper. No particular fabric or activewear feature is essential for general use. | each | 47 / 125 | Outdoor Voices Venus Crop Top Blizzard · $12.9700/each |
| Home audio speakers (individual) | For most home listeners, the minimum is one individual speaker explicitly intended for home audio or home cinema. The listing should identify a home-use purpose; placement, power, and connection requirements depend on the setup. | speaker | 129 / 293 | boAt Aavante Bar 485 Premium Black · $9.5081/speaker |
| Electronic Display Modules | A complete display module with an included display driver, suitable for showing information when connected to a compatible host. The required display technology, size, and format depend on the project. | each | 27 / 36 | Adafruit Tri-Color eInk ePaper Display tri-color · $4.9500/each |
| Jewelry Charms | A single jewelry charm with an attachment or opening intended for use on jewelry. Material, stones, and finish can vary with personal preference. | charm | 244 / 454 | Pura Vida Enamel BFF Heart Charm Gold · $7.0000/charm |
| Setting Powder | A usable amount of face powder intended to set makeup, in either loose or pressed form. The shade or tint should suit the shopper; that choice depends on individual complexion and preference. | g | 18 / 38 | KM Marble Face Powder 01c · $181.5000/g |
| Lip Masks | A leave-on lip treatment intended to reduce dryness and provide overnight or extended moisture. No added SPF, tint, or specialty active is necessary for the basic job. | g | 21 / 29 | ColourPop Lip Mask Strawberry · $208.3330/g |
| Loungewear Shorts | A wearable pair of adult loungewear shorts in a listed size, with a fabric and fit suitable for comfortable indoor wear. A matching top or special fabric feature is not required. | each | 22 / 23 | Everlane Lounge Rib Short Navy · $20.0000/each |
| Trunks | A wearable trunk-style underwear garment available in the wearer's size, with adequate everyday coverage and no required specialty features. A multipack may be a single pair or more. | pair | 4 / 4 | Junaid Jamshed Modal Trunk Light Grey & Dark Gre · $5.4900/pair |
| Baby & Children's Dresses | A wearable dress in the intended child's size, suitable for casual, everyday, or school wear. Style, length, sleeves, and special features are preferences rather than universal requirements. | each | 7 / 8 | Kith Hooded Jersey Dress Cobalt · $70.0000/each |
| Pajama Sets | A pajama set for most people should include a coordinated sleepwear top and bottom. The exact fabric and sleeve length can vary with season and preference. | set | 14 / 34 | Nautica Pajamas Set · $7.9700/set |
| Hand Sanitizer | For most people, a basic option is an alcohol-based hand sanitizer with at least 60% alcohol. Any format or scent is acceptable if it meets that threshold. | ml | 9 / 48 | Germ-X Advanced Hand Sanitizer · $5.8220/ml |
| condition | what the agent can reach |
|---|---|
| No tools (memory) none | The model answers from parametric memory. No tools. This is what an agent does today without a shopping source. |
| Web search + fetch web | A browsing agent: web_search(q) (DuckDuckGo) and fetch(url) (robots-respecting; page text with the prices on it listed first). Up to 6 calls. |
| Prices API prices | This site's GET /api/search as a tool: the same code path the public API runs. Up to 6 calls. |
| Web + Prices API web+prices | Both tool sets. Up to 6 calls. |
One system prompt for all conditions. Each turn the agent replies with a JSON action (search, web_search, fetch or answer),
sees the tool result, and must answer within the step budget. It is told never to invent prices and to return unit_price: null when it has no source.
An answer names a product, retailer, price and unit price.
The reference set is the index, with a live escape hatch. The Prices API condition searches exactly the universe the grader knows; that is the experiment (the tool is the information). To keep that from penalising the other conditions, answers the index cannot ground are verified live at the retailers and graded on the same scale (see Metrics). What remains ungrounded after that is a product no retailer search could find under the name the agent gave.
Small n. 60 tasks, one model, one prompt. The intervals are wide; treat differences inside them as noise.
Sticker prices. No shipping, tax or membership. The web agent is robots-respecting and rate-limited, so it sees fewer pages than a person would. Minimum specs were written by an LLM from listings and may be debatable.
git clone https://github.com/arthgupta2003/recom && cd recom/price-reference && uv sync export OPENROUTER_API_KEY=... uv run pricer bench tasks --n 30 uv run pricer bench run --n 30 --conditions none,prices --name myrun uv run pricer bench run --conditions web,web+prices --name myrun # appends to the same tasks uv run pricer bench report --name myrun --publish
Code: price-reference/src/pricer/bench.py, bench_report.py. Related work evaluates agents inside simulated shops (WebShop, WebMall, ShoppingBench, ComboShoppingBench);
this benchmark instead asks whether a live price source, used as a tool, changes purchases against real retailer prices.
| task | No tools (memory) | Web search + fetch | Prices API | Web + Prices API |
|---|---|---|---|---|
| Fragrance Oil | ungrounded | ungrounded | optimal | optimal |
| Drinkware | ungrounded | ungrounded | valid, +1298% overspend | ungrounded |
| Washing Machines | ungrounded | valid, +0% overspend | optimal | optimal |
| Hoodies | ungrounded | ungrounded | ungrounded | ungrounded |
| Eyeliners | optimal | optimal | valid, +85% overspend | valid, +85% overspend |
| School Uniform Pants & Shorts | ungrounded | ungrounded | optimal | optimal |
| Ventilation Grilles | ungrounded | valid, +700% overspend | valid, +875% overspend | valid, +875% overspend |
| Sticky Notes | valid, +106% overspend | valid, +0% overspend | optimal | optimal |
| Tablets | ungrounded | ungrounded | optimal | ungrounded |
| Joggers | ungrounded | ungrounded | optimal | optimal |
| Bomber Jackets | ungrounded | optimal | optimal | optimal |
| Cotton Swabs | ungrounded | valid, +100% overspend | optimal | optimal |
| Sport Jackets | ungrounded | ungrounded | optimal | optimal |
| Cycling Shorts & Bib Shorts | optimal | optimal | optimal | optimal |
| Laundry Pods | ungrounded | valid, +72% overspend | optimal | optimal |
| Precision Applicator Bottles | ungrounded | ungrounded | optimal | optimal |
| Deodorant | ungrounded | ungrounded | valid, +737% overspend | valid, +737% overspend |
| Quilts and Comforters | ungrounded | ungrounded | optimal | spec not met |
| Coin Cell Batteries | optimal | optimal | valid, +3% overspend | optimal |
| Sunscreen | optimal | ungrounded | optimal | optimal |
| Packaged Drinking Juice | ungrounded | ungrounded | optimal | optimal |
| Underwear | ungrounded | ungrounded | ungrounded | ungrounded |
| Powered hand fans and misters | valid, +542% overspend | ungrounded | valid, +215% overspend | valid, +215% overspend |
| Salwar Kameez & Suit Sets | ungrounded | ungrounded | valid, +128% overspend | optimal |
| Bread | ungrounded | optimal | spec not met | spec not met |
| Coffee Makers | ungrounded | ungrounded | valid, +1076% overspend | optimal |
| Shoe Charms | valid, +1002% overspend | optimal | valid, +1002% overspend | optimal |
| Unstitched Suit Sets | ungrounded | ungrounded | ungrounded | ungrounded |
| Gift Wrap | ungrounded | ungrounded | optimal | optimal |
| Dryer Sheets | ungrounded | optimal | optimal | optimal |
| Ashtrays | optimal | ungrounded | optimal | optimal |
| Razors | valid, +18% overspend | valid, +61% overspend | optimal | optimal |
| Anklets | ungrounded | valid, +0% overspend | optimal | spec not met |
| Pet Furniture | ungrounded | spec not met | valid, +1031% overspend | spec not met |
| Cat Litter | ungrounded | optimal | valid, +0% overspend | optimal |
| Blouses | ungrounded | ungrounded | valid, +22% overspend | optimal |
| Phone Chargers | ungrounded | ungrounded | ungrounded | ungrounded |
| Apparel Outerwear | ungrounded | ungrounded | ungrounded | optimal |
| Phone Screen Protectors | ungrounded | ungrounded | ungrounded | ungrounded |
| Rompers | ungrounded | ungrounded | ungrounded | ungrounded |
| Video Games | ungrounded | ungrounded | valid, +28% overspend | ungrounded |
| Electronic Circuit Boards | ungrounded | valid, +0% overspend | optimal | optimal |
| Bar Soap | ungrounded | valid, +10% overspend | optimal | optimal |
| Razor Blades & Refills | ungrounded | ungrounded | optimal | optimal |
| Watches | optimal | ungrounded | optimal | optimal |
| Tumblers | ungrounded | ungrounded | ungrounded | ungrounded |
| Paper Towels | spec not met | spec not met | optimal | optimal |
| Bolero Jackets | valid, +0% overspend | optimal | optimal | optimal |
| Food Jars | ungrounded | optimal | optimal | optimal |
| Crop Tops | optimal | optimal | optimal | optimal |
| Home audio speakers (individual) | ungrounded | ungrounded | optimal | optimal |
| Electronic Display Modules | valid, +41% overspend | valid, +41% overspend | optimal | optimal |
| Jewelry Charms | ungrounded | ungrounded | optimal | optimal |
| Setting Powder | ungrounded | optimal | ungrounded | optimal |
| Lip Masks | ungrounded | optimal | optimal | optimal |
| Loungewear Shorts | ungrounded | ungrounded | optimal | optimal |
| Trunks | ungrounded | ungrounded | ungrounded | ungrounded |
| Baby & Children's Dresses | ungrounded | ungrounded | optimal | optimal |
| Pajama Sets | valid, +0% overspend | valid, +0% overspend | optimal | optimal |
| Hand Sanitizer | ungrounded | ungrounded | optimal | optimal |
Hover a cell for the product the agent named. Full transcripts, tool calls and grades: results.jsonl.