robots.txt for AI: Let GPTBot and ClaudeBot In
A lot of sites are invisible to AI answers for a boring reason: an old robots.txt rule, written years ago to stop scrapers, also blocks the bots that write those answers. The site owner never notices, because nothing looks broken.
If GPTBot, ClaudeBot or PerplexityBot can't crawl your pages, the matching engine can't cite you. Fixing that is a few lines of config. It is also a real decision, because letting these bots in means letting your content be used in AI answers.
Which AI crawlers do I allow?
Four cover the major engines: GPTBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and Google-Extended for Google's AI features. Each is a separate user-agent, so you list them one by one. Miss one and you drop out of that engine's answers.
The rules to add
# robots.txt — welcome the AI answer engines
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# Still keep private areas out, for every bot
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Sitemap: https://yourdomain.com/sitemap.xmlEach AI bot gets its own block so a later wildcard Disallow can't sweep it up by accident. And your admin and checkout paths stay closed. Welcoming AI crawlers and protecting private pages sit happily together.
Check the bots can actually read you
A robots.txt rule is only half the job. If your content only appears after heavy JavaScript runs, a crawler that doesn't render scripts still sees a blank page. After editing robots.txt, confirm your key facts sit in the raw HTML. A one-line fetch shows what a simple crawler gets:
curl -s https://yourdomain.com/your-page | grep "your key fact"If the fact doesn't show, move it into server-rendered HTML so the bot can reach it.
Should you allow them at all?
For most businesses chasing visibility, yes. Being missing from AI answers usually costs more than being summarised in them, especially since answer engines send small numbers of high-intent visitors. A publisher guarding a paywalled archive might weigh it differently. Either way, decide on purpose rather than leaving an old rule in charge.
Key takeaways
- List GPTBot, ClaudeBot, PerplexityBot and Google-Extended one by one in robots.txt.
- You can welcome AI bots and still block /admin and /checkout.
- Fetch the raw HTML to confirm your facts are visible without JavaScript.
- Whether to allow AI crawlers is a business call, so make it deliberately.
Unsure which bots your site blocks today? A Creative Nexus website audit flags crawlability gaps for Google and the AI engines together.
Frequently asked questions
What does robots.txt do for AI?
It tells web crawlers which parts of your site they may read. For AI, it decides whether bots like GPTBot and ClaudeBot can access your pages, which in turn decides whether those engines can cite you.
Which AI crawlers should I allow?
GPTBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and Google-Extended for Google's AI features. Each is a separate user-agent, so list them individually to stay eligible for citation.
Does allowing AI bots expose my whole site?
No. You can allow AI crawlers on public content while still blocking private paths like /admin or /checkout for all bots. Welcoming crawlers and protecting sensitive areas are separate rules.
How do I know if I'm blocking AI crawlers?
Look in your robots.txt for Disallow rules matching GPTBot, ClaudeBot, PerplexityBot, or a broad wildcard. A website audit can flag these crawlability gaps automatically across Google and the AI engines.
Is a robots.txt rule enough for AI to read a page?
Not always. If content only appears after JavaScript runs, a crawler that doesn't render scripts sees nothing. Confirm your key facts are in the raw server HTML, not only in the rendered view.
What is Google-Extended?
A separate user-agent that controls whether Google may use your content for its generative AI features, such as AI Overviews and Gemini, independently of normal Google Search crawling.
Will blocking AI bots protect my content?
It stops those engines citing you, which usually costs more visibility than it protects. Some paywalled publishers block them, but most businesses gain more from being citable.
How do I test what a crawler sees?
Fetch the page with a simple request such as curl and search the raw HTML for your key fact. If the fact only appears after scripts run, a basic crawler may miss it.
Do I need a sitemap line?
It helps. A Sitemap directive points crawlers to your full URL list, improving discovery for classic search and AI engines alike. Keep the sitemap current and submitted in Search Console.
Should every business allow AI crawlers?
Most should, since being absent from AI answers is usually a bigger risk than being summarised, and the referrals are high-intent. It remains a business decision, so weigh visibility against any need to protect content.
Want this done for your site?
Run a free audit and see exactly what to fix for Google and AI search.
