AEO in 2026: Why Blocking AI Crawlers Backfires
Sites aiming to appear in ChatGPT, Claude and Perplexity answers need a new robots.txt approach. Here's how AEO differs from SEO and how to manage crawlers.
As of 2026, search behavior is shifting: users increasingly get direct answers from ChatGPT, Claude, or Perplexity instead of clicking through Google rankings. This has given rise to Answer Engine Optimization (AEO), a discipline that inverts classic SEO instinct by requiring sites to actively welcome AI crawlers rather than block them. The piece stresses that each provider runs separate bots for training, search indexing, and live fetch — blocking one, like GPTBot, does nothing to stop others like OAI-SearchBot.
It also clarifies that robots.txt, standardized as RFC 9309, is a voluntary convention with no legal enforcement; while major crawlers generally comply, some rotate user-agents and IPs to evade directives, meaning real blocking must happen at the CDN/WAF layer. A previously circulated claim that rel="canonical" tags signal 'source of truth' to AI crawlers is debunked as unverified. Instead, server-rendered HTML, schema.org structured data, and the emerging llms.txt convention are presented as the actual levers that matter.
A key practical warning is that CDN-level bot management (e.g., Cloudflare) can silently override a site's actual robots.txt settings, causing unintended blocks. The article closes with a sample per-bot robots.txt configuration and a tip to audit server logs for 403 errors against AI search bots to catch accidental visibility loss.