Short answer: Crawl budget is the set of URLs Google can and wants to crawl on your site in a given time. It depends on crawl capacity (how much your server can handle) and crawl demand (how popular and fresh your pages are). Most small sites never need to worry about it. Large or fast-changing sites optimise it by cutting duplicate and parameter URLs, fixing redirects and errors, speeding up the server and keeping sitemaps clean.
This guide is for developers, SEOs and owners of larger sites (online shops, classifieds, news and directory sites) who want to know whether crawl budget is really their problem and what to do if it is. I will explain how Google decides how much to crawl, how to measure it in Search Console and server logs, what wastes it, and the fixes I apply in order. The calculator below gives you a quick reality check with your own numbers.
What crawl budget means
Googlebot cannot fetch every URL on the web as often as it would like, so it decides how much to crawl on each site. Google’s crawl budget guide for large sites defines crawl budget as the set of URLs Google can and wants to crawl, made up of two parts:
| Part | What drives it | What you control |
|---|---|---|
| Crawl capacity limit (can crawl) | How many parallel connections and how quickly Googlebot can crawl without overloading your server. It rises when the site responds fast and drops with slow responses, 5xx errors and 429 rate-limiting | Server speed, hosting resources, error rate, CDN and caching |
| Crawl demand (wants to crawl) | How many URLs Google thinks you have (perceived inventory), how popular they are, and how stale its copy is | Which URLs exist and are linked, duplicates, internal links, sitemaps with accurate lastmod |
Two points I always make to clients. First, crawl budget covers every URL Googlebot fetches, not just pages: JavaScript, CSS, images, API calls and redirecting URLs all use it. Second, being crawled more is not a ranking factor in itself. Crawling is what gets new and updated pages seen; it does not make an indexed page rank higher. The broader process is explained in my guide to crawling and indexing.
Does your site need to care?
Google is direct about this: its guide is aimed at very large sites (roughly a million or more unique pages changing weekly), medium or larger sites (around 10,000 or more pages changing daily), and sites with a large share of URLs stuck in “Discovered, currently not indexed”. A 200-page business site or a blog with a few hundred posts usually gets crawled thoroughly without any effort.
The exception I see often in Pakistan and elsewhere: small sites that generate huge numbers of URLs by accident. A WooCommerce shop with 300 products can expose tens of thousands of filter, sort and add-to-cart URLs. A classifieds or property site can create a page for every search combination. Those sites have a crawl budget problem caused by their own URL inventory, not by their size.
Crawl coverage calculator
How to read it: take requests per day from Settings, Crawl stats in Search Console, and estimate the useful share from the response and file type breakdowns (or from server logs). This is a rough planning estimate, not how Google schedules crawling: popular pages are recrawled often and others rarely. Its value is showing how much crawling goes to URLs you do not care about.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
How to measure crawling
Search Console Crawl Stats
Open Settings, then Crawl stats. The Crawl Stats report shows total crawl requests, total download size, average response time, and breakdowns by response code, file type, crawl purpose (discovery or refresh) and Googlebot type, plus host status. What I look for:
- Average response time creeping up, which usually means less crawling soon after.
- A high share of 301, 404 or 5xx responses, each a request not spent on a real page.
- Host status warnings for robots.txt fetching, DNS or server connectivity.
- Discovery vs refresh: if almost everything is refresh on a site that publishes daily, new URLs are not being found quickly.
Server logs
Logs show every request Googlebot actually made, URL by URL. Filter for verified Googlebot, then group by directory, parameter and status code. This is where you find the 40,000 requests a month going to ?orderby=price. I explain the process in my guide to log file analysis.
The Page indexing report
A growing “Discovered, currently not indexed” count is the clearest crawl budget symptom in Search Console. My guide to the Search Console coverage report explains how to read it alongside crawl data.
What wastes crawl budget
| Waste | Typical example | Fix |
|---|---|---|
| Faceted navigation and filters | /shoes?colour=black&size=9&sort=price | Block non-useful combinations in robots.txt, keep only valuable filter pages crawlable and linked |
| Session IDs and tracking parameters | ?sessionid=, ?utm_source= in internal links | Never use them in internal links; strip them server-side |
| Duplicates | http and https, www and non-www, trailing slash variants | 301 redirect to one version; consistent internal links |
| Infinite spaces | Calendars, endless pagination, internal search result pages | robots.txt disallow for search; limit calendar depth |
| Redirect chains | A to B to C after several migrations | Point links and redirects to the final URL |
| Soft 404s and thin pages | Empty categories, expired listings returning 200 | Return 404 or 410, or add real content |
| Slow or failing server | Shared hosting at peak times, 5xx errors | Better hosting, caching, CDN, fix errors |
| Hacked or spam URLs | Thousands of injected pages | Clean up, return 404/410, secure the site |
For shops, my guide to faceted navigation goes into which filter pages to keep, and crawl traps covers infinite URL spaces in more depth.
How I optimise crawl budget, in order
- Shrink the URL inventory. Find URL patterns Google crawls that you would never want in search results, and stop generating or linking them. This has the biggest effect on most sites.
- Block what must exist but should not be crawled. Use robots.txt disallow rules for internal search, add-to-cart, sort and session URLs. Test changes with the robots.txt tester before deploying, because one wrong line can block a whole section.
- Consolidate duplicates. Redirect protocol and host variants, fix trailing slash inconsistencies, and use canonicals for near-duplicates you have to keep.
- Return the right status codes. Removed pages should return 404 or 410, not 200 and not a redirect to the homepage. Remove soft 404s.
- Fix redirect chains and broken internal links. Each hop is an extra request.
- Speed up the server. Faster responses raise the capacity limit. Caching, a CDN and enough server resources all help; check response times with the PageSpeed checker and in Crawl Stats.
- Keep sitemaps clean and current. List only canonical, indexable 200 URLs, and use accurate lastmod dates so Google knows what changed.
- Strengthen internal links to important pages. Pages that are well linked and shallow in the structure get crawled more. See my guide to click depth.
Things that do not help
- noindex to save crawl budget. Google still has to crawl the page to see the noindex, so it saves nothing on crawling.
- Temporarily blocking sections in robots.txt to shift budget elsewhere. Google’s guide advises against this; the budget does not move over unless you are already capacity-limited.
- Crawl-delay in robots.txt. Googlebot ignores it.
- The Indexing API for normal pages. It is only supported for job postings and livestream video pages.
Worked example: robots.txt rules for a WooCommerce shop
User-agent: *
Disallow: /?s=
Disallow: /search/
Disallow: /*?add-to-cart=
Disallow: /*?orderby=
Disallow: /*&orderby=
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Sitemap: https://www.example.com/sitemap_index.xmlThis keeps product, category and useful filter pages crawlable while stopping the most common crawl waste on WordPress shops. Adapt the patterns to your own URLs and confirm in your logs that the blocked patterns were actually being crawled before you add rules.
When Google crawls too much
Occasionally the problem is the opposite: Googlebot (or more often, other bots) overloads a small server. Google’s guide says 5xx and 429 responses make Googlebot reduce its crawl rate, so returning 503 or 429 briefly during a genuine overload works, but leaving it in place for days will start to affect indexing. The longer-term fixes are caching, more server resources and blocking abusive non-Google crawlers. If AI crawlers are the cause, my guide to AI crawlers and robots.txt shows how to manage them separately.
Frequently asked questions
Is crawl budget a ranking factor?
No. Crawling decides whether and how quickly Google sees your pages and updates. It does not boost rankings for pages that are already indexed. A crawl problem can still cost you traffic by delaying new or changed pages.
How do I check my crawl budget?
Search Console’s Crawl Stats report (Settings, then Crawl stats) shows daily crawl requests, response times and breakdowns by status code and file type. For URL-level detail, analyse your server logs filtered to verified Googlebot.
Does noindex save crawl budget?
No. Google has to crawl a page to see its noindex tag. To stop URLs being crawled, block them in robots.txt or stop generating and linking them.
Do small websites need to optimise crawl budget?
Usually not. Sites with a few thousand pages or fewer are normally crawled well. The exception is a small site that generates huge numbers of parameter, filter or search URLs, which is worth fixing regardless of size.
How do I increase my crawl budget?
Raise capacity with a faster, more reliable server, and raise demand by making important pages popular and well linked. Then cut waste so the existing budget goes to the URLs that matter.
Sources
- Google Search Central, “Crawl budget management for large sites” (accessed 2026).
- Search Console Help, “Crawl Stats report” (accessed 2026).
- HTTP Archive, “Web Almanac 2024: SEO chapter” (2024).
If you run a large shop or listings site and your new pages take weeks to appear in Google, my technical SEO service includes log analysis and a crawl waste audit.
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.