Free Robots.txt Generator

A robots.txt generator writes the plain text file that tells crawlers which parts of your site they may visit. The free Robots.txt Generator above builds one with presets for search engines and AI crawlers. It runs in your browser, so nothing you type is uploaded or stored anywhere.

The 2024 Web Almanac found that 83.9% of mobile sites returned a 200 status for robots.txt, so most sites already publish one and a wrong rule can affect many pages at once.

How to use the Robots.txt Generator

1. Choose a preset for search engines or AI crawlers, or start from the one closest to your needs.
2. Add or adjust the rules for the paths you want crawlers to avoid.
3. Add your sitemap address if the tool offers a field for it.
4. Copy the finished text and save it as a file named exactly robots.txt.
5. Upload it to the root of your domain, so it loads at yourdomain.com/robots.txt.
6. Test it with the Robots.txt Tester and AI Crawler Checker before you rely on it.

How to read the results

The output is a short list of rules. Each group starts with a User-agent line naming the crawler, followed by Allow or Disallow lines for paths. A Sitemap line points to your sitemap.

• User-agent: * means the rules apply to all crawlers that have no group of their own.
• Disallow: /private/ asks crawlers to stay out of that folder.
• Disallow: with nothing after it means nothing is blocked for that crawler.
• Disallow: / blocks the entire site. Check carefully before you publish it.

A good file is short, blocks only areas with no search value, and never blocks the scripts or styles a page needs to display.

What to fix before you publish

• A blanket Disallow. A single Disallow: / left over from development can hide the whole site from crawling. Remove it on live sites.
• Blocking pages you want indexed. Robots.txt controls crawling, not indexing. To keep a page out of results, use a noindex directive and leave the page crawlable so the directive can be read.
• Treating it as security. The file is public and advisory. Do not list secret folders in it. Protect private content with a login.
• Wrong location or name. It must be a plain text file called robots.txt at the root, not inside a folder.
• Careless AI crawler blocks. Decide deliberately which AI crawlers you allow. Blocking one affects whether that service can fetch your pages, so match the choice to your goals.

A simple example

This is an illustrative example. A small shop wants search engines to crawl everything except its basket and checkout pages, and wants to list its sitemap. The generated file would have one group for all crawlers, two Disallow lines for those paths, and one Sitemap line. Nothing else is needed. Short files are easier to audit later.

Why it matters for SEO

Crawlers have a limited amount of attention for any site, and a clean robots.txt keeps them focused on the pages that matter. Faceted filters, internal search results and basket pages can create huge numbers of near-duplicate URLs that add nothing to search. Keeping crawlers away from them is a sensible use of the file. The risk runs the other way too: one careless line can stop crawlers reaching whole sections, and the damage is easy to miss because the site still looks fine to visitors. That is why the safe routine is to generate, review line by line, publish, and then test.

Keep a copy of your previous file before replacing it, and note the date of each change. If traffic drops after an edit, you will then know exactly what changed.

Limits to know about

The generator makes text from the options you pick. It does not look at your site, so it cannot know which of your folders matter. Rules are followed voluntarily by well-behaved crawlers, and other bots may ignore them. Rules also differ slightly between crawlers in how they read edge cases. Once the file is live, test real URLs rather than assuming the rules work. To create the sitemap you reference, the XML Sitemap Generator can build one from a pasted list of URLs.

Questions about the Robots.txt Generator

Does robots.txt stop a page appearing in Google?

Not reliably. It asks crawlers not to fetch a URL, but a blocked URL can still be shown if other pages link to it. To keep a page out of results, use noindex and keep it crawlable.

Should I block AI crawlers in robots.txt?

That is a business choice. Blocking asks those crawlers not to fetch your pages, which may limit how those services use your content. Allowing them may help visibility there. The presets give you a starting point to adjust.

Where do I put the robots.txt file?

Place it in the root of the domain so it loads at the top level. Each subdomain needs its own file. A robots.txt inside a subfolder is not read as the site file.

Back to all free technical SEO tools or the full toolkit. If you would rather have someone check the whole site, see my technical SEO services or ask for a free audit.