How to set up robots.txt for AI crawlers
To let AI engines cite you, allow their crawlers in robots.txt: add Allow rules for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, keep private paths blocked, then confirm your facts appear in raw HTML.
Add the rules
In your robots.txt, give each AI crawler its own block so a later wildcard can't sweep it up: 'User-agent: GPTBot / Allow: /', and the same for ClaudeBot, PerplexityBot and Google-Extended. Keep a 'User-agent: * / Disallow: /admin/' block for private areas, and add your Sitemap line.
Check the bots can actually read you
A rule is only half the job. If your content loads only after heavy JavaScript, a crawler that doesn't render scripts sees nothing. Fetch the page raw and search for a key fact; if it isn't there, move that fact into server-rendered HTML.
Decide on purpose
Allowing these bots means your content can appear in AI answers. For most businesses chasing visibility that is the goal, since absence from AI answers usually costs more than being summarised. Make it a deliberate choice, not an old anti-scraper rule left in place.
Frequently asked
GPTBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and Google-Extended for Google's AI features. List each separately, since missing one drops you from that engine.
No. You can allow crawling of public content while still blocking paths like /admin or /checkout for all bots. The two rules coexist.
Fetch the page with a simple request and search the raw HTML for a key fact. If it only appears after scripts run, a basic crawler may miss it.
Grade your site against our 250-point AEO/GEO checklist and see exactly what to fix, free, no signup.