AI crawler robots.txt checker.
Paste your robots.txt to see which AI crawlers may read your pages, and which rule decides it.
Questions.
Should I block AI crawlers?
It depends on what you want. Blocking training crawlers such as GPTBot or CCBot keeps your pages out of future models. Blocking search and user agents such as OAI-SearchBot, Claude-User or Perplexity-User also keeps you out of AI answers that could send you visitors.
How does this checker decide?
It follows the robots.txt rules crawlers use: a crawler obeys the group that names it, else the * group, and the longest matching Allow or Disallow path wins. An empty Disallow allows everything.
Does robots.txt stop every bot?
No. It is a request, not a lock. The crawlers listed here say they follow it, but others may not.
Is my robots.txt sent anywhere?
No. The check runs in your browser and nothing you paste leaves the page.
See which channels actually pay
These tools estimate. Virling measures: it links every visit to the payment it led to, so revenue per visitor and return come from your real numbers. 14 days free.