How does it decide Allowed or Blocked?

It tests the homepage path against your robots.txt using Google's matching rules and the most specific user-agent group.

What if I have no robots.txt?

Then every crawler is allowed by default.

It covers the major documented ones. New crawlers appear often, so recheck occasionally.

Analyzer

See which AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) your robots.txt allows and which it blocks.

Try:

The page is fetched by our server (it can't be done from your browser), analysed, and not stored.

Allowing crawlers is half the job. IndexBoost submits your URLs to search engines so they get discovered.

Context

01
Training and search are different bots
Blocking GPTBot stops training use, while OAI-SearchBot controls ChatGPT search results. Decide separately for each purpose.
02
Blocking search bots costs visibility
If you want to be cited in AI answers, make sure the search and user-agent bots are allowed.
03
robots.txt is voluntary
Well-behaved crawlers obey it. If you need enforcement, use firewall or CDN rules.