No. Google does not support the crawl-delay directive in robots.txt and ignores it. Some other crawlers, such as Bing, do honour it. If crawlers are straining your server, you need a different approach for Google, such as improving server capacity or temporarily returning an error status.
What crawl-delay does
The directive asks a bot to wait a set number of seconds between requests:
User-agent: bingbot
Crawl-delay: 5It is a request, not an enforcement, and each crawler decides whether to follow it.
Why Google ignores it
Google adjusts its crawl rate automatically based on how quickly your server responds. If responses slow down or errors rise, Googlebot backs off. That is why crawl-delay is not part of its supported robots.txt rules.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
What to do if Googlebot is overloading your site
- Check server capacity and caching first. Slow responses often cause the problem.
- Reduce unnecessary crawlable URLs, such as endless filters, using robots.txt rules or better site structure.
- In a genuine emergency, return a temporary 503 or 429 status, which tells Googlebot to slow down. Do not leave it in place for long.
Google also retired its old crawl rate limiter tool in Search Console, so the main levers now are server health and URL control. My guide to crawl budget explains how to reduce wasted crawling.
Other crawlers
Bing documents support for crawl-delay and also offers its own crawl controls in Bing Webmaster Tools. Behaviour among other bots varies, and many AI crawlers have their own rules, which I cover in AI crawlers explained.
Writing it safely
If you decide to add crawl-delay for a specific bot, place it inside that bot’s own group and keep the number small. Create the file with my robots.txt generator and verify it in the robots.txt tester.
Frequently asked questions
Will crawl-delay slow down only the bot I name?
Yes, if you place it inside that bot’s group. A rule under the wildcard group applies to every bot that reads it.
Can too long a delay hurt me?
For crawlers that obey it, yes. A large value can cut how many pages get crawled in a day.
Next steps
If crawler traffic is causing real server strain, contact me here and I will help you find the cause.
More in this series (robots.txt questions): start with Free Robots.txt Generator, then explore:
- How to Add a Sitemap to Robots.txt
- Robots.txt Wildcards: How * and $ Work With Examples
- Indexed, Though Blocked by Robots.txt: Why It Happens and How to Fix It
- WordPress Robots.txt: Where It Is, How to Edit It and What to Include
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.