Analyzer

Free Robots.txt Tester

Test whether a URL is allowed or blocked for a given crawler under a robots.txt file, and see the exact rule responsible.

Paste a robots.txt, or load the live one for the host above.

Crawlable? Get it submitted

Once robots.txt lets crawlers in, IndexBoost submits your URLs to speed up discovery.

Get started free

Context

How robots.txt matching works

  1. 01

    The longest matching rule wins

    /shop/sale allows beat a shorter /shop/ disallow. If lengths tie, Allow wins.

  2. 02

    Crawlers use one group

    A crawler follows the most specific user-agent group that matches it, not all groups combined.

  3. 03

    Wildcards and $

    * matches any characters and $ anchors the end of the URL. Example: Disallow: /*.pdf$

Common questions

← All free tools