Robots.txt generator

Create a robots.txt file to control how bots crawl a website.

  • SEO
  • 100% free
  • No sign-up
  • Runs in your browser
Robots.txt generator
seconds
0 = no delay. Google ignores this rule.
One path per line.
Exceptions inside the blocked paths, one per line.
Reset

What is the robots.txt generator?

The robots.txt generator creates the file that tells crawlers which parts of a website they may visit. Block admin areas, carts and internal search pages, allow exceptions, add the location of your sitemap and, optionally, block the bots that collect data to train AI models.

robots.txt controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. Use a noindex tag for pages that must stay out of search engines.

How to use the robots.txt generator

  1. 1Choose whether all robots are allowed or blocked.
  2. 2List the paths to block and the exceptions, one per line, and add your sitemap URL.
  3. 3Download robots.txt and upload it to the root of your website.

Features

Blocked and allowed paths

Disallow and Allow rules, one path per line.

Sitemap

Point crawlers to your XML sitemap.

Block AI crawlers

One switch adds rules for GPTBot, CCBot, Google-Extended, ClaudeBot, PerplexityBot and more.

Download

Get a ready robots.txt file.

Frequently asked questions

Where do I put robots.txt?

In the root folder of the domain, so it opens at https://example.com/robots.txt. Each subdomain needs its own file.

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing. Use a noindex meta tag, and let Google crawl the page so it can see it.

Should I block CSS and JavaScript files?

No. Google needs them to render the page and judge it correctly.

Does Google follow Crawl-delay?

No, Google ignores it. Bing, Yandex and some other crawlers respect it.