What Is Robots.txt? Syntax, Examples and SEO

Short answer: Robots.txt is a plain text file at the root of your site (yourdomain.com/robots.txt) that tells search engine crawlers which URLs they may or may not crawl. It matters because it controls how bots spend their time on your site and can accidentally block your whole site from Google. It is not a way to hide pages: to keep a page out of search results, use a noindex tag or password protection instead.

83.9%of mobile sites returned a valid (200) robots.txt file in 2024, so most sites have one, whether or not anyone checks itHTTP Archive Web Almanac, 2024

500 KiBGoogle reads only the first 500 KiB of a robots.txt file and generally caches it for up to 24 hoursGoogle Search Central

2.7%of mobile sites named GPTBot in robots.txt in 2024, the most-mentioned AI crawlerHTTP Archive Web Almanac, 2024

This guide is for site owners, WordPress users and developers who want to read, write or fix a robots.txt file with confidence. You will learn what each rule does, how Google interprets it, ready-to-use examples, the mistakes I find most often in audits, and how to decide whether to block AI crawlers.

What is robots.txt?

Robots.txt is a text file that follows the Robots Exclusion Protocol. Well-behaved crawlers such as Googlebot and Bingbot request it before crawling a site and follow its rules. Google describes its main use as managing crawler traffic to your site, and says plainly that it is not a mechanism for keeping a web page out of Google (Google’s introduction to robots.txt).

That distinction causes most of the confusion I see. Robots.txt controls crawling (whether a bot fetches a URL). It does not control indexing (whether a URL appears in results). If other sites link to a blocked URL, Google can still index the address and show it without a description.

Robots.txt vs meta robots vs X-Robots-Tag

Tool	Where it lives	Controls	Use it for
robots.txt	One file at the site root	Crawling of URL paths	Keeping bots out of low-value areas: internal search, filters, admin, carts
Meta robots tag	In each page’s HTML head	Indexing and link following for that page	Keeping a page out of results with `noindex`
X-Robots-Tag	HTTP response header	Indexing for any file type	Keeping PDFs, images or other non-HTML files out of results

One rule follows from this: if you want Google to see a noindex tag, the page must be crawlable. Blocking it in robots.txt hides the noindex. My guide to the robots meta tag explains the page-level options.

How to check your robots.txt file

Visit https://yourdomain.com/robots.txt in a browser. If you see rules, the file exists; a 404 means there is none, which Google treats as “crawl everything”.
In Google Search Console, open Settings and then the robots.txt report. It shows the version Google last fetched, when, and any parsing problems.
Test specific URLs against your rules with my free robots.txt tester.

Each subdomain and protocol needs its own file. Rules at www.example.com/robots.txt do not apply to shop.example.com.

Not sure if this is hurting your site?

Get the fixes that matter most, in priority order.

Find my blockers

Robots.txt syntax explained

A file is made of groups. Each group starts with one or more User-agent lines and is followed by rules. Google supports four fields:

User-agent: which crawler the group applies to. * means all crawlers. A crawler follows only the most specific group that matches its name.
Disallow: a path the crawler should not fetch. Disallow: / blocks everything; an empty Disallow: blocks nothing.
Allow: an exception inside a disallowed path.
Sitemap: the full URL of your XML sitemap. It can appear anywhere and you can list several.

How Google reads the rules

Paths are case-sensitive: /Admin/ and /admin/ are different.
* matches any sequence of characters and $ marks the end of a URL, so Disallow: /*.pdf$ blocks URLs ending in .pdf.
When Allow and Disallow both match, the most specific (longest) rule wins; if they are equally specific, the less restrictive rule wins.
Crawl-delay is not supported by Google. Bing does support it.
Noindex in robots.txt has not worked in Google since 2019. Use the meta tag instead.

What happens when the file is unavailable

According to Google’s robots.txt specification, a 4xx response (except 429) is treated as if there were no file, so everything may be crawled. A 5xx server error makes Google stop crawling the site for the first 12 hours, then fall back on the last good copy for up to 30 days. A firewall or CDN that intermittently returns 5xx for robots.txt can therefore slow down crawling of the whole site.

Robots.txt examples you can copy

Allow everything

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

A sensible WordPress file

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://www.example.com/sitemap_index.xml

This keeps bots out of the admin area and internal search results while allowing the AJAX file many themes need. Do not block /wp-content/ or /wp-includes/: Google needs CSS and JavaScript to render pages.

An online shop with filters

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?orderby=
Disallow: /*?filter_

Sitemap: https://www.example.com/sitemap.xml

Filter and sort parameters can create thousands of near-duplicate URLs. Blocking them saves crawl effort for product and category pages. See my guide to faceted navigation before you block filters that people actually search for.

Block a staging site

User-agent: *
Disallow: /

Better still, protect staging with a password. And make sure this file never gets copied to the live site.

Should you block AI crawlers?

AI companies run several crawlers with different jobs. Blocking one does not block the others, so decide bot by bot. OpenAI, for example, says its GPTBot and OAI-SearchBot settings are independent: GPTBot relates to model training, OAI-SearchBot to appearing in ChatGPT search.

User-agent	Owner	Purpose	Effect of blocking
GPTBot	OpenAI	Collects content for model training	Content not used for training; does not remove you from ChatGPT search results
OAI-SearchBot	OpenAI	Powers ChatGPT search	Your pages may stop appearing as ChatGPT search sources
Google-Extended	Google	A control token for use in Gemini training and grounding	Does not affect Google Search rankings or crawling by Googlebot
PerplexityBot	Perplexity	Indexes pages for Perplexity answers	Fewer citations in Perplexity
CCBot	Common Crawl	Open web archive often used to train models	Content kept out of future Common Crawl datasets

My usual advice for businesses that want visibility is to allow search-focused bots and decide on training bots based on how much your content is your product. A publisher with paid content may block training crawlers; a local service business usually benefits from being cited. More detail is in my guide to AI crawlers and robots.txt and the wider topic of generative engine optimisation.

Should this URL be blocked in robots.txt?

Robots.txt or something else?

Is the content private or sensitive (customer data, staging, internal documents)?

Do you want the URL kept out of Google’s search results?

Is it a large set of low-value URLs, such as internal search results, filters, sort orders or session parameters?

Is it a duplicate of another page that should get the credit?

Result: use authentication, not robots.txt. Robots.txt is public and only a request; anyone can read it and badly behaved bots ignore it. Put the content behind a login or remove it.

Result: leave it crawlable. If you want it in search, do not block it. Make sure it is in your sitemap and linked internally.

Result: use noindex, and keep it crawlable. Add a meta robots noindex tag (or X-Robots-Tag for files). Once it has dropped out of results, you can add a robots.txt block for large URL sets if crawl load is a real problem.

Result: use a canonical tag or redirect. Point the duplicate to the main version so signals consolidate. Blocking it in robots.txt would stop Google seeing the canonical.

Common robots.txt mistakes I find in audits

Disallow: / left on after launch. The single most damaging error; it usually comes from a staging copy.
Blocking CSS and JavaScript. Old WordPress advice blocked /wp-includes/. Google then cannot render the page properly.
Blocking pages that carry noindex. Google never sees the tag, so the URL can stay indexed as “Indexed, though blocked by robots.txt”.
Listing secret paths. Disallow: /private-client-files/ tells everyone where to look.
Wrong location or case. The file must be at the root, named robots.txt in lower case.
Security plugins or CDNs blocking bots. On Pakistani hosting I often see firewall rules that return errors to Googlebot or to all non-browser traffic, which looks like a robots.txt problem but is not.
Forgetting the sitemap line. Not fatal, but an easy win for discovery.

Robots.txt mistakes are one of the first things I check in a crawling and indexing review. On large sites, it is also a tool for managing crawl budget.

How to create and edit robots.txt

Write the rules in a plain text editor, or generate a starting file with my free robots.txt generator.
Save as UTF-8 with the exact name robots.txt.
Upload it to the root of the domain. In WordPress, Yoast SEO and Rank Math let you edit it from the dashboard; without a physical file, WordPress serves a virtual one.
Test important URLs against the new rules before and after publishing.
Check Search Console’s robots.txt report a day later to confirm Google fetched the new version.

Frequently asked questions

Do I need a robots.txt file?

Not strictly. Without one, crawlers assume they can crawl everything, which is fine for many small sites. It becomes useful when you want to keep bots away from internal search, filters, admin areas or specific crawlers.

Does robots.txt stop a page from being indexed?

No. It stops crawling. A blocked URL can still be indexed if other pages link to it, usually shown without a description. Use noindex or password protection to keep a page out of results.

Where is the robots.txt file in WordPress?

At yourdomain.com/robots.txt. WordPress generates a virtual file if none exists; SEO plugins such as Yoast and Rank Math let you edit it, or you can upload a physical file to the site root.

How long does Google take to see robots.txt changes?

Google generally caches the file for up to 24 hours. You can ask for a faster refresh from the robots.txt report in Search Console.

Is robots.txt a security tool?

No. It is public and voluntary. Malicious bots ignore it, and listing private folders can draw attention to them.

Does blocking GPTBot remove my site from ChatGPT?

Blocking GPTBot stops OpenAI using your content for training. ChatGPT search uses a separate crawler, OAI-SearchBot, so manage the two separately.

Sources

Google Search Central, “Introduction to robots.txt” (last updated 2025).
Google Search Central, “How Google interprets the robots.txt specification”.
HTTP Archive, “Web Almanac 2024: SEO chapter” (2024).
OpenAI, “Overview of OpenAI crawlers” (2026).

If you are not sure whether your robots.txt, server responses or noindex tags are holding pages back, my technical SEO services include a full crawl and indexing review. You can also ask me to check your file.