What is robots.txt?
Robots.txt is a text file at your site's root that tells crawlers which parts of the site they may or may not crawl.
In plain English
It lives at yourdomain.com/robots.txt and issues “allow”/“disallow” rules per crawler. It controls crawling, not indexing, a disallowed page can still be indexed if linked elsewhere, so use a noindex tag to truly keep a page out of results.
Why it matters for AI search
Robots.txt now also governs AI crawlers (like GPTBot and Google-Extended). It's where you decide whether AI engines may read your content to train on or cite, a deliberate choice for AI visibility.
Frequently asked
It blocks crawling, but a blocked page can still appear in results if other sites link to it. To remove it, use a noindex meta tag instead.
Only if you don't want AI engines using your content. To be cited by ChatGPT and others, allow their crawlers (e.g. GPTBot, Google-Extended).
Grade your site against our 250-point AEO/GEO checklist and see exactly what to fix, free, no signup.