Generator
Free Robots.txt Generator
Build a robots.txt file in a few clicks — block private folders, point crawlers to your sitemap, and decide which AI bots can read your site.
Default for all crawlers
Quick add:
Sub-paths inside a blocked folder that should still be crawled.
One per line, full URL.
Ignored by Google; Bing and Yandex respect it.
AI crawlers
User-agent: *
Disallow:
Upload it to the root of your domain so it's reachable at https://yourdomain.com/robots.txt.
robots.txt lets crawlers in. Indexing gets them there faster
A clean robots.txt tells search engines what they're allowed to crawl — it doesn't make them come sooner. IndexBoost submits your new and updated URLs directly to Google, Bing and IndexNow so they get crawled without the wait.
Get started freeContext
Before you upload
- 01
Blocking crawling isn't the same as blocking indexing
If other sites link to a URL you've disallowed, Google can still list it — just without a description. To keep a page out of results, let it be crawled and add a noindex tag instead.
- 02
Never block your CSS and JavaScript
Search engines render pages much like a browser does. If they can't load your stylesheets and scripts, they may see a broken layout and misjudge the page.
- 03
Training bots and answer bots are different
Several AI companies run one crawler for training models and another that fetches pages to answer a live question. Blocking only the training bots keeps you visible in AI answers without contributing to training data.
- 04
Test after every change
One stray 'Disallow: /' can wipe a whole site from search. After uploading, open the file in a browser and check the robots.txt report in Google Search Console.