Preparing a website for AI platforms starts with one technical decision most sites have never made deliberately: which AI bots are actually allowed in. AI search visits grew 42.8% year over year, from 15.6 billion to 27.4 billion between Q1 2025 and Q1 2026 — and a site that’s silently blocking the wrong crawlers, whether through a default robots.txt or an aggressive CDN bot-blocking preset, is opting out of that traffic without ever deciding to.
Two Kinds of AI Bots, and Why the Difference Matters
The major AI companies now run separate bots for two different jobs, and treating them as one category is the single most common mistake in this setup:
- Training crawlers — GPTBot, ClaudeBot, CCBot, Google-Extended — collect pages to build the dataset a model is trained on. Your content becomes part of the model’s weights, usually with no attribution and no referral traffic back to the site.
- Search and retrieval bots — OAI-SearchBot, Claude-SearchBot, PerplexityBot — index pages specifically so the AI assistant can cite them in a live answer and send a visitor back to the source. These are the bots that generate the referral traffic this whole exercise is about.
Because these run as genuinely separate bots at the major providers, a site can opt out of feeding model training while staying fully eligible for AI-search citations — blocking GPTBot doesn’t have to mean blocking OAI-SearchBot too. That distinction is the entire basis of a sensible 2026 crawler policy.
Why the Traffic Is Worth Configuring For
The volume is still small relative to traditional organic search, but the growth curve is the actual story. Digiday data showed ChatGPT referral traffic increasing 52% year-over-year in a recent measured window, while Gemini referral traffic grew 388% in the same period. ChatGPT alone accounts for roughly 87.4% of all AI referral traffic today, which means a robots.txt rule that blocks OpenAI’s fetch agents closes off the largest single AI discovery channel available right now.
A Practical Setup, Not a Theoretical One
- Set robots.txt deliberately. Allow the search/retrieval agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) to stay eligible for citations. Block training-only crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) if training-data control matters more to you than the marginal training-related visibility.
- Check the CDN separately. Cloudflare, Fastly, and similar platforms ship aggressive bot-blocking presets that catch AI crawlers alongside genuine scrapers. A robots.txt that correctly allows a bot means nothing if the CDN is silently returning a 403 before the request even reaches it.
- Add an llms.txt file as a guide, not a gate. An llms.txt points AI agents to a site’s best, most citation-worthy pages — but it carries no enforcement power. Crawlers can read it and ignore it entirely. Access control still belongs to robots.txt and the server or CDN, not llms.txt.
- Track AI referrals as their own channel in analytics. If ChatGPT-driven visits aren’t showing up as a distinct segment, there’s no way to know whether the crawler policy above is actually converting into traffic.
Diagnosing the Two Failure Modes
When AI referral traffic is flat, the cause is almost always one of two distinct problems, and they need different fixes:
- Crawlers aren’t visiting at all. Check server logs (or an analytics tool’s bot-traffic section) for GPTBot, ClaudeBot, and PerplexityBot activity. If they’re absent, the problem is almost certainly robots.txt or a CDN bot rule — an access problem, not a content problem.
- Crawlers visit, but no citations follow. If server logs show real crawl activity but referral traffic stays at zero, the content itself isn’t being judged citable. That’s a content-quality problem — see our guide on getting cited by ChatGPT and Perplexity for what actually earns a citation once a crawler can reach the page.
A Simple Monthly Habit
Scan server logs monthly for AI crawler activity on the pages that matter most. Quarterly, run 20–30 category-relevant prompts directly in ChatGPT, Claude, and Perplexity to check whether the brand actually gets cited. This is the same diagnostic loop covered in more depth in our overview of generative engine optimization and our guide to tracking whether AI search engines are mentioning your brand at all.
The Bottom Line
Most sites haven’t made an active decision about AI crawler access — whatever their robots.txt or CDN happened to ship with is the decision they’re currently living with. Given that AI search visits grew nearly 43% year over year and referral traffic from platforms like Gemini is growing at tripledigit rates, that default setting is worth checking deliberately rather than by accident.