CodesBeautify

robots.txt Generator

Build a robots.txt with disallowed paths, an optional block for AI crawlers, and your sitemap.

robots.txtGenerated
User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml
Runs locally in your browser

About the robots.txt Generator

A robots.txt tells search engines and other crawlers which parts of a site to skip and where the sitemap is. This generator writes one from a few choices: the paths to disallow, whether to block all crawling (useful for a staging site), an optional crawl delay, the sitemap URL, and a set of rules blocking the common AI training crawlers (GPTBot, CCBot and Google-Extended) for sites that opt out of that use.

A Disallow value is matched as a prefix of the URL path and it is case-sensitive. Disallow: /admin therefore covers /admin, /admin/users and also /administrators, which is rarely what anyone means; add the trailing slash to keep it to the directory. Disallow: / shuts out the whole site, an empty Disallow: blocks nothing at all, and every rule belongs to the User-agent line above it.

The trap is assuming this hides a page. Blocking a URL stops crawling but not indexing, and a blocked page that other sites link to can still surface as a bare link with no snippet, precisely because the crawler is forbidden to fetch it and so never sees the noindex tag you added. To take a page out of results, allow crawling and serve noindex, or put the page behind authentication.

Remember that robots.txt is advisory: well-behaved crawlers follow it, but it is not access control and does not hide pages from people. Keeping a staging site out of search results is the usual reason to write one from scratch. The Sitemap Generator builds the sitemap the file points to, and the Meta Tag Generator writes the noindex tag when that is what you actually need.

How to use

  1. Enter the paths to disallow and your sitemap URL.
  2. Tick AI-crawler blocking or block-all if needed.
  3. Save the output as robots.txt in your site root.

Common questions

Does Disallow hide pages from Google?
It stops crawling but not indexing of already-known URLs. Use a noindex meta tag to remove pages from results.
Which AI crawlers are blocked?
GPTBot (OpenAI), CCBot (Common Crawl) and Google-Extended (Google AI training).
Is robots.txt a security measure?
No. It is a request that crawlers may ignore. Protect private content with authentication.