The AI Crawler Roster: Who to Allow in robots.txt and Why

Citation starts with access. If GPTBot or ClaudeBot is blocked, your GEO ceiling is zero regardless of content quality, and blocks hide in CDN rules as often as in robots.txt. For a commercial brand the trade is simple: visibility inside AI answers is worth more than theoretical content protection, so allow retrieval agents on public pages and verify in logs that they actually crawl.

Get into the retrieval pool

Generative engines compose answers from a small pool of retrieved sources per topic. Run your real buyer prompts monthly, log every cited domain, and treat that list as your outreach plan. A placement inside the pool outperforms dozens of generic links.

Publish quotable claims

Models quote specifics: numbers, definitions, named comparisons. Vague service pages never get cited; pages with a defensible statistic get lifted repeatedly. Give every money page at least one claim worth quoting verbatim.

Key takeaways

Keep entity signals identical everywhere

When your site, profiles, and third-party mentions describe the brand differently, models hedge and cite someone else. Lock one canonical description of what you do and for whom, then deploy it everywhere without creative variation.

In practice

The practical failure mode we see most is a brand doing everything right on content while a single blocked crawler caps the entire program at zero, and the block hiding in a CDN rule rather than robots.txt. So the first thing we do is enumerate the current agent roster, OpenAI's GPTBot, OAI-SearchBot, and ChatGPT-User; Anthropic's ClaudeBot and Claude-User; Perplexity's two agents; Google-Extended and Applebot-Extended, and verify each is allowed across robots, CDN, and firewall layers. Then we confirm in server logs that the agents actually crawl, because allowing them in the rules and having them arrive are different things. For a commercial brand the trade is simple: visibility inside AI answers is worth more than theoretical content protection on public pages. Getting access right is unglamorous and it is the precondition for everything else in GEO, which is why we check it before anything else.