From September 15, 2026, Cloudflare's stated policy is to block AI training and "agent" crawlers by default on sites that display ads, allowing only search-oriented AI crawlers unless the site owner changes the setting. That means a site can be perfectly open in its robots.txt yet still return blocks to AI crawlers at the network edge — a gap a robots.txt checker alone will never show you.
Why this causes confusion
Two independent layers now decide crawler access: your robots.txt (which you control directly) and your CDN/WAF rules (which a provider like Cloudflare may change on your behalf). When AI-referral traffic suddenly drops, owners often inspect robots.txt, find nothing wrong, and miss that the block is happening at the edge.
What to check
Run a live crawler check on your homepage: it compares a normal browser response to each crawler's user-agent and flags cases where the server answers a bot differently — the signature of edge-level blocking. Then review your CDN's bot-management settings. For each crawler's verified identity, see the crawler directory.
This is a technical note about crawler reachability, not advice about content-licensing or crawling policy.