You can write the best content on the internet and still be invisible to AI search — not because the content is weak, but because a single misconfigured file is quietly telling AI crawlers to stay out. Lesson 3.4 of Ahrefs’ AEO Course, taught by Sam Oh, tackles the unglamorous but critical layer underneath everything else in this course: if AI can’t access your site in the first place, none of your content, keyword, or mention work matters.
Here’s the full breakdown of robots.txt, the AI-specific crawler bots, and llms.txt — and how to actually configure them.
Why This Is the Foundation, Not an Afterthought
Every lesson so far in this series has assumed AI platforms can actually see your content. That assumption breaks more often than most site owners realize. Many sites built or last audited before AI crawlers existed have robots.txt files that either say nothing about AI bots (leaving default behavior ambiguous) or, worse, inherit an overly aggressive “block everything” rule from a theme, plugin, or bot-protection layer that was never meant to catch AI crawlers — it was meant to stop malicious scrapers, and AI bots got caught in the net by accident.
This matters because AI crawler traffic isn’t a minor footnote anymore. GPTBot’s request volume alone surged over 300% year-over-year according to Cloudflare’s crawler analysis, jumping from a minor player to one of the most active bots crawling the web. If your technical setup is quietly blocking that traffic, you’re opting out of AI visibility without ever making that decision on purpose.
Understanding the Different AI Bots (They’re Not All the Same)
The lesson’s core technical insight is that “AI crawler” isn’t one thing — different bots from the same company serve different purposes, and treating them identically is a mistake. The distinction that matters most is training vs. retrieval:
- Training crawlers (like OpenAI’s GPTBot, Anthropic’s ClaudeBot, and Google-Extended) collect content to train or improve future model versions. Blocking these prevents your content from shaping how the model “thinks” — but it doesn’t necessarily affect whether you get cited in a live answer today.
- Retrieval/search crawlers (like OpenAI’s OAI-SearchBot, PerplexityBot, and ClaudeBot’s search-specific variants) fetch content in real time to answer a specific user query. Blocking these directly removes you from citation eligibility — if the bot can’t fetch your page right now, it can’t cite it right now.
This split gives you a genuine strategic choice rather than an all-or-nothing decision. A site that doesn’t want its proprietary content used to train a competitor’s model, but still wants to be cited when ChatGPT users search for related topics, can disallow GPTBot while explicitly allowing OAI-SearchBot. That’s a meaningfully different policy than either “block everything” or “allow everything,” and most default configurations don’t express it.
The Robots.txt Reality Check
Robots.txt itself isn’t new — it’s a decades-old convention formalized as an official internet standard, and every major AI crawler that behaves well respects it, the same way Googlebot does. The lesson’s practical guidance boils down to three checks:
- Check your live robots.txt directly — don’t assume it’s fine because you configured it once years ago. Staging configs, CDN rules, and security plugins can silently override your intent.
- Add explicit rules per AI bot rather than relying on a generic wildcard rule to cover them. A default
User-agent: *withDisallow: /blocks every AI crawler along with everything else, even if you only meant to block one. - Remember robots.txt only controls crawling, not removal. It’s a request not to fetch a page — it doesn’t retroactively remove content already absorbed into a trained model, and a blocked page can technically still surface if other sources reference it elsewhere.
It’s also worth being honest about the limits of access alone. Simply allowing AI bots to crawl your site doesn’t guarantee citations — plenty of sites that explicitly allow GPTBot still see zero citations from ChatGPT, because access without genuinely citable, well-structured content — the same “answer the question clearly” discipline covered in how to optimize content for AI search engines — doesn’t get you cited on its own.
What About llms.txt?
llms.txt is the newer, more experimental piece of this lesson. Proposed in September 2024 by Jeremy Howard of Answer.AI, it’s a plain-text (typically Markdown) file placed at your site’s root that gives language models a curated, cleanly formatted map of your most important content — effectively a sitemap written for models instead of crawlers.
The honest caveat the lesson is careful to include: llms.txt is not an official, ratified standard, and as of now no major AI platform has confirmed it reads or relies on the file for citations. Adoption across the web remains low. That doesn’t mean it’s worthless — it costs very little to implement, and if it becomes more widely supported later, sites that already have one in place get an early advantage. Treat it as a low-cost hedge on the future, not a lever that will move your AI visibility today.
A Practical Checklist From This Lesson
- Pull up your live robots.txt right now and read it line by line — don’t rely on memory of how you configured it originally.
- Add explicit, named rules for AI bots instead of relying on a generic wildcard. Cover GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, and PerplexityBot at minimum.
- Decide your training vs. retrieval policy deliberately. Disallowing a training bot doesn’t have to mean disallowing that same company’s retrieval/search bot.
- Don’t treat robots.txt as a privacy tool. It stops future crawling; it doesn’t erase content already indexed elsewhere or already used in training.
- Check server logs for AI bot activity. If you see zero requests from GPTBot, ClaudeBot, or PerplexityBot, that’s a strong signal something upstream — a CDN rule, a security plugin, a misconfigured robots.txt — is blocking them.
- Add an llms.txt file as a low-cost hedge, understanding it’s an early-stage convention rather than a confirmed ranking or citation factor today.
- Pair technical access with genuinely citable content. Being crawlable is necessary but not sufficient — it only creates the opportunity to be cited, it doesn’t guarantee it.
Final Thought
It’s easy to skip the technical lesson in a course like this — it’s less exciting than keyword research or brand mentions, and the payoff is invisible until something’s broken. But this lesson is really a foundation check for every other lesson in the series. Great content and a strong mention strategy can’t overcome a robots.txt file quietly telling AI crawlers to turn around at the door. Before investing further in content or citation-building work, it’s worth five minutes to go check — not assume, check — that the door is actually open.