Robots.txt Generator

A robots.txt generator builds the small plain-text file that lives at the root of your domain and tells search engine crawlers which paths they may fetch and which they should skip. This free generator gives you a form-based interface so you never have to remember the exact syntax: pick the crawlers you want to target, add allow and disallow rules for specific paths, optionally reference your sitemap, and the tool writes a clean, standards-compliant file you can copy or download. The format is simple but easy to get wrong by hand. Each block starts with a User-agent line (for example, User-agent: * for all bots, or User-agent: Googlebot for one), followed by Disallow and Allow lines listing paths. Paths are matched as prefixes, so Disallow: /admin blocks /admin and everything beneath it. A blank Disallow line means nothing is blocked, which is how you explicitly allow a crawler full access. The single most important thing to understand is what robots.txt does not do: it does not keep a page out of Google's index. It asks well-behaved crawlers not to fetch a path, but if that page is linked from elsewhere, Google can still list the URL in results without a description. If you genuinely need a page kept out of search, use a noindex meta tag or an X-Robots-Tag header instead, and do not also disallow the page, because a blocked crawler can never see the noindex tag. The file must sit at the exact root, such as https://example.com/robots.txt; crawlers will not look for it in a subfolder. The Sitemap directive, by contrast, takes an absolute URL and belongs on its own line, typically at the bottom. This generator is handy when launching a site, blocking staging or admin areas, or adding rules to keep AI crawlers like GPTBot from training on your content. Everything runs in your browser, so nothing about your site is sent to a server.