Analyzer
Free Robots.txt Tester
Test whether a URL is allowed or blocked for a given crawler under a robots.txt file, and see the exact rule responsible.
Paste a robots.txt, or load the live one for the host above.
Crawlable? Get it submitted
Once robots.txt lets crawlers in, IndexBoost submits your URLs to speed up discovery.
Get started freeContext
How robots.txt matching works
- 01
The longest matching rule wins
/shop/sale allows beat a shorter /shop/ disallow. If lengths tie, Allow wins.
- 02
Crawlers use one group
A crawler follows the most specific user-agent group that matches it, not all groups combined.
- 03
Wildcards and $
* matches any characters and $ anchors the end of the URL. Example: Disallow: /*.pdf$