Free Robots.txt Generator
Build a robots.txt file in a few clicks — block private folders, point crawlers to your sitemap, and decide which AI bots can read your site.
Default for all crawlers
Quick add:
Sub-paths inside a blocked folder that should still be crawled.
One per line, full URL.
Ignored by Google; Bing and Yandex respect it.
AI crawlers
User-agent: *
Disallow:
Upload it to the root of your domain so it's reachable at https://yourdomain.com/robots.txt.
robots.txt lets crawlers in. Indexing gets them there faster
A clean robots.txt tells search engines what they're allowed to crawl — it doesn't make them come sooner. IndexBoost submits your new and updated URLs directly to Google, Bing and IndexNow so they get crawled without the wait.
Get started freeContext
Before you upload
Blocking crawling isn't the same as blocking indexing
If other sites link to a URL you've disallowed, Google can still list it — just without a description. To keep a page out of results, let it be crawled and add a noindex tag instead.
Never block your CSS and JavaScript
Search engines render pages much like a browser does. If they can't load your stylesheets and scripts, they may see a broken layout and misjudge the page.
Training bots and answer bots are different
Several AI companies run one crawler for training models and another that fetches pages to answer a live question. Blocking only the training bots keeps you visible in AI answers without contributing to training data.
Test after every change
One stray 'Disallow: /' can wipe a whole site from search. After uploading, open the file in a browser and check the robots.txt report in Google Search Console.
FAQ