In robots.txt, the asterisk (*) matches any sequence of characters and the dollar sign ($) marks the end of a URL. Together they let you block patterns such as parameters or file types without listing every URL. Google also picks the most specific matching rule, so the order of lines is less important than their length.
The asterisk wildcard
The * stands for any run of characters, including none. This rule blocks every URL containing a sort parameter:
User-agent: *
Disallow: /*?sort=The dollar sign end marker
The $ means the URL must end at that point. This blocks PDF files but not a page whose address merely contains .pdf somewhere in the middle:
User-agent: *
Disallow: /*.pdf$Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
How conflicting rules are resolved
When an Allow and a Disallow both match a URL, Google follows the rule with the longest matching path. If they are equal in length, the less restrictive rule (Allow) wins. This lets you block a folder and then open one file inside it:
Disallow: /private/
Allow: /private/public-guide.htmlPaths are case sensitive
/Photos/ and /photos/ are different to a crawler. Match the exact case your site uses.
Test before you publish
A single misplaced wildcard can block far more than intended. Draft the rules with my robots.txt generator and try specific URLs in the robots.txt tester.
When wildcards help crawl efficiency
On large sites, wildcards are a practical way to keep crawlers away from endless filter or session URLs. This ties directly into crawl budget. Remember that blocking crawling is different from removing a page from search, which I explain in noindex vs canonical vs robots.txt.
Frequently asked questions
Do all search engines support wildcards?
Google and Bing support * and $. Smaller or older crawlers may not, so keep critical rules simple.
Do I need a wildcard at the end of a path?
Not usually. Rules already match by prefix, so Disallow: /folder/ covers everything inside it.
Next steps
If you are unsure which patterns are safe to block on your site, contact me here for a technical review.
More in this series (robots.txt questions): start with Free Robots.txt Generator, then explore:
- How to Add a Sitemap to Robots.txt
- Indexed, Though Blocked by Robots.txt: Why It Happens and How to Fix It
- WordPress Robots.txt: Where It Is, How to Edit It and What to Include
- Does Google Respect Crawl-Delay in Robots.txt?
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.