Amazon locked its robots.txt against OpenAI's crawlers; a few weeks later, its share of ChatGPT citations collapsed and Walmart took its place. Blocking AI crawlers protects your content — but the bill comes due in LLM visibility, exactly where your buyers now ask their questions. A field note on the monetization-vs-citations trade-off, and the only sane way to settle it: crawler by crawler, never "AI" as a monolith.
- Amazon restricted nearly 50 AI user-agents in its robots.txt (including all three OpenAI crawlers) → its ChatGPT citation share dropped; Walmart, which stayed open, overtook it.
- Blocking a retrieval crawler = dropping out of the answers, not just out of future model training.
- Pay-per-crawl (Cloudflare, July 2025): Allow / Charge / Block per crawler, HTTP 402 response — and AI crawlers blocked by default for new customers.
- The right decision grain is the crawler, not "AI" — training, retrieval, and agents don't carry the same GEO cost.
- The arbitration rule: if content is your product → Charge or Block; if content is your distribution → Allow.
What happened between Amazon and Walmart in ChatGPT?
The summary fits in one sentence: Amazon shut the door on OpenAI's crawlers, and ChatGPT started recommending Walmart. In October 2025, Amazon dominated ChatGPT citations among major retailers. Between late 2025 and January 2026, its robots.txt kept hardening — ChatGPT-User, OAI-SearchBot, then dozens of other bots — until nearly 50 AI user-agents were restricted, including all three OpenAI crawlers. By November, its ChatGPT citation share was falling sharply. Walmart, which stayed open and signed an Instant Checkout partnership with OpenAI, filled the void and took the lead on ChatGPT.
Sources: Amazon's robots.txt (Modern Retail, Search Engine Land), Q1 2026 ChatGPT citation data, working paper from OpenAI's Economic Research team.
One detail that matters: on Google AI Overviews, Amazon keeps a comfortable lead. The block only cost where it applies — every platform builds its citations differently, which is exactly why we track AI citation fragmentation across platforms as a topic in its own right.
Why does blocking AI crawlers tank LLM citations?
Because a ChatGPT or Perplexity answer isn't generated from the model's memory alone: on commercial queries, the model fires off real-time retrieval, reads whatever sources it can access, then cites what it managed to read. Blocking the retrieval crawler means pulling yourself off the candidate source list at query time. The model isn't punishing anyone — it simply cites whoever stays readable.
Walmart didn't get better in November 2025. It just stayed accessible while Amazon went unreadable. The demand signal you've been building is worthless if, at retrieval time, the crawler hits a 403.
Amazon protected its catalog; Walmart took its citations. robots.txt is no longer a line of technical hygiene — it's a distribution decision.
That's what we measure at Rankfeed: citation rate per model is the immediate tell for this kind of infrastructure decision.
Does Cloudflare's pay-per-crawl change the trade-off?
It makes it explicit — which is already a lot. On July 1, 2025, Cloudflare announced pay-per-crawl: a model where every AI crawler request gets either the content (HTTP 200, with declared payment intent) or an HTTP 402 Payment Required response carrying the price. Three options per crawler: Allow (free access), Charge (payment at the domain's set price), Block (refusal, no payment option). Crawler authentication relies on cryptographic signatures to counter user-agent spoofing, and Cloudflare acts as merchant of record.
Same day, another shift, more structural: Cloudflare now blocks AI crawlers by default for its new customers. The web flipped from an opt-out model (everything crawlable unless refused) to an opt-in model (nothing crawlable without consent).
If your site moved behind Cloudflare after July 2025, you may be blocking AI crawlers without ever having decided to. Audit your AI Crawl Control settings and your logs before wondering why your citation rate is flat.
On the revenue side, let's be honest: the unit price of a crawl is low, and monetization only becomes meaningful at very high crawl volume — a large-publisher profile, not a SaaS one. The full equation, GEO cost included:
Should you block all AI crawlers, or decide crawler by crawler?
Crawler by crawler, no exceptions. "Blocking AI" is a category that doesn't exist technically: a training crawler, a search crawler, and an agent acting on a user's behalf don't play the same role — and don't carry the same cost when you shut them out.
The operational sequence, as we run it:
- 011 — InventoryServer logs + AI Crawl Control: which AI crawlers actually come through, how often, on which pages. Deciding without this inventory is deciding blind.
- 022 — ClassifySort each user-agent into one of three families: training, retrieval, agent. Each vendor's docs (OpenAI, Anthropic, Perplexity) spell out what each bot does.
- 033 — Set a citation baselineCitation rate per model across your query cluster, BEFORE touching robots.txt. Without a baseline, there's no attributing a drop to the block.
- 044 — Decide, then monitorApply the policy crawler by crawler and track the effect on citations week after week — exactly the kind of loop our How it works page describes.
And a necessary correction: llms.txt restricts no crawler whatsoever. It's a navigation file, not access control — we documented the myth and reality of llms.txt in detail. Real control lives in robots.txt (a voluntary directive) and at the WAF/CDN level (forced enforcement).
How do you weigh monetization vs GEO visibility for your business?
The whole grid fits in one question: is your content the product, or the distribution? A publisher sells its content — every uncompensated crawl is a dead loss, and Charge is a rational position. A B2B SaaS or an e-commerce brand sells something else: content is just the vehicle that gets the brand into AI answers. Closing it off means closing your own acquisition channel — and with the rise of agentic commerce through ChatGPT, blocking ChatGPT-User amounts to turning away a customer who showed up holding their credit card.
Weighing monetization vs visibility
Amazon's case isn't even a strict "mistake": Amazon can afford to lose ChatGPT citations — its brand generates its own demand. You probably can't. Every lost citation is a recommendation handed to a competitor — and retrieval keeps no memory of your former position.
FAQ
Keep reading
- llms.txt: myth vs reality — what the file actually does (and doesn't).
- AI citation fragmentation across platforms — why Amazon drops on ChatGPT but holds on AI Overviews.
- Agentic commerce in ChatGPT and the ACP protocol — the stakes behind blocking ChatGPT-User.
Your crawl policy is a GEO decision. Set your citation baseline across ChatGPT, Claude, Grok, and DeepSeek, then steer the effect of every change — How it works walks through the measure → action loop, and Pricing starts with no commitment.