SEO tools

Robots.txt Generator and Tester

Build a robots.txt file for your site, then test it: see whether a URL is allowed or blocked for Googlebot, Bingbot or AI crawlers, and which rule decides.

Runs in your browser
Free, no sign-up

This tool runs in your browser and needs JavaScript. Please enable it to use the tool.

Start from

Allow every crawler everywhereThe safest choice for most sites

Block these paths (Disallow, one per line)

* matches anything, $ marks the end: /*.pdf$ blocks every PDF.

But allow (Allow, optional exceptions)

Block AI training crawlers . Google and Bing search are not affected.

Sitemap URL (one per line)

Crawl-delay (seconds, optional; Google ignores it)

robots.txt content

Crawler

User-agent

URLs or paths to test (one per line)

Results for

How to use the Robots.txt Generator and Tester

1
Pick a starting point (allow everything, a standard site, WordPress) and adjust the paths to block.
2
Add your sitemap URL and, if you like, block AI training crawlers such as GPTBot and CCBot.
3
Download robots.txt and upload it to the root of your site (https://your-site.com/robots.txt), then test URLs in the Test tab.

Frequently asked questions

Does robots.txt remove a page from Google?

No. It stops crawling, not indexing: a blocked page can still appear in results if other sites link to it. To keep a page out of Google, allow crawling and add a noindex meta tag instead.

Which rule wins when Allow and Disallow both match?

The most specific (longest) rule wins. If both are equally long, Allow wins. This tester uses the same rules as Google.

Should I block AI crawlers?

Blocking GPTBot, CCBot, ClaudeBot or Google-Extended stops your content being collected to train AI models. It does not affect your ranking in Google or Bing search.

Where does the file go?

At the root of the domain: https://example.com/robots.txt. A robots.txt in a sub-folder is ignored, and each subdomain needs its own.