Check which AI crawlers your site lets in.
Free check: read any site's robots.txt and llms.txt and see which AI crawlers — GPTBot, ClaudeBot, PerplexityBot and friends — are allowed or blocked.
Reads two public files. Nothing else on your site is fetched.
- 01
Enter your domain
Crawlpatrol reads your robots.txt and your llms.txt, the two files written for exactly this.
- 02
Get a verdict per crawler
Fifteen AI crawlers, each marked allowed, blocked, partly blocked, or never mentioned.
- 03
Decide on purpose
Training crawlers and answer crawlers are separate tokens: you can opt out of one and stay citable in the other.
About Crawlpatrol
Most sites block AI crawlers by accident. A robots.txt written in 2019, a template copied from a boilerplate, one Disallow line inherited from a staging config — and ChatGPT, Claude and Perplexity quietly stop being able to cite you.
Crawlpatrol reads the two files every site publishes for exactly this purpose, robots.txt and llms.txt, and gives a verdict per crawler. Nothing else on the site is fetched.
Questions
- Should I block AI crawlers?
- Usually not all of them. Training crawlers (GPTBot, ClaudeBot, Google-Extended) and answer crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) are separate tokens, so you can stay citable in AI answers while opting out of training.
- What does “not mentioned” mean?
- Neither that crawler nor the wildcard group appears in your robots.txt. Under the robots standard that means allowed — allowed by omission rather than on purpose.
- What is llms.txt?
- A plain-text file at the root of your site that tells language models what you do and which pages matter. It is not part of the robots standard, and no crawler is required to read it, but it is cheap to publish.
Crawlpatrol answers one question. Seiton answers the rest every morning.
Connect Stripe, analytics, Search Console and Sentry read-only. Seiton tells you what changed overnight, how much money is at risk, and what deserves attention today.
Get my first briefing14-day free trial. No credit card.