llms.txt vs robots.txt
robots.txt tells crawlers which URLs they may access; llms.txt is an emerging file that gives AI models a curated, plain-text map of your most important content to help them understand and cite you.
The difference
robots.txt is a gate, it allows or disallows crawling (including AI bots like GPTBot and Google-Extended). llms.txt is a guide, a Markdown file at your root that summarises your site and links your key pages, making it easier for LLMs to grasp what you offer and quote you accurately.
Use them together
Use robots.txt to decide whether AI crawlers may read you at all; use llms.txt to help the ones you welcome understand your best content. One controls access, the other improves comprehension.
Frequently asked
It's a community proposal gaining adoption, not yet a universal standard, but it's low-cost to add and forward-looking for AI visibility.
It can block specific AI crawlers (e.g. GPTBot, Google-Extended) from reading your content, if that's your choice.
Grade your site against our 250-point AEO/GEO checklist and see exactly what to fix, free, no signup.