Robots.txt User-Agent Groups: Which Rules Does Googlebot Follow?

A crawler follows only the single most specific user-agent group that matches its name and ignores the rest. If you have a group for Googlebot and another for the wildcard, Googlebot reads only the Googlebot group. Rules are not added together across different groups, which is the most common source of confusion.

How a group works

A group begins with one or more User-agent lines followed by the Allow and Disallow rules that apply to them. A new User-agent line after rules starts a new group.

User-agent: *
Disallow: /search/

User-agent: Googlebot
Disallow: /tmp/

What happens in this example

Googlebot follows its own group and may crawl /search/, because the wildcard group does not apply to it. Every other crawler follows the wildcard group and is blocked from /search/ but not from /tmp/. If you want Googlebot to avoid both, repeat the rule inside its group.

Not sure if this is hurting your site?

Get the fixes that matter most, in priority order.

Find my blockers

Specific agents and fallbacks

Some Google crawlers fall back to the Googlebot group when they have no group of their own, while others can be controlled separately. When you target a specific crawler, name it exactly as its documentation does. For the AI crawlers, see AI crawlers explained.

Repeated groups for the same agent

If the same user agent appears in several groups, Google combines them into one set of rules. Keeping each agent in a single group is still cleaner and easier to maintain.

Practical rules to follow

Put shared rules in the wildcard group.
If you add a specific group, copy over any shared rules it needs.
Keep one group per crawler.
Leave a blank line between groups.

Build and verify your groups with the robots.txt generator and the tester.

Frequently asked questions

Does order matter between groups?

No. The crawler picks the most specific match regardless of order.

Does the Sitemap line belong to a group?

No. It stands alone, as explained in how to add a sitemap to robots.txt.

Next steps

If your rules behave unexpectedly, contact me here for a technical review.

More in this series (robots.txt testing questions): start with Robots.txt Tester and AI Crawler Checker, then explore: