Short answer: Crawl demand is how much Google wants to crawl your site’s URLs. Google says it depends on the URLs it knows about (perceived inventory), how popular those URLs are, and how stale its stored copies are. You influence it by cutting low-value and duplicate URLs, linking and promoting the pages that matter, updating content genuinely, and sending accurate sitemap lastmod dates.
This guide is for site owners and SEOs who see new or updated pages sitting uncrawled for weeks, or who run larger sites (shops, classifieds, news, directories) where Googlebot seems to spend its time in the wrong places. You will learn what crawl demand is, how it differs from crawl capacity, how to read it in Search Console and your logs, and the levers that actually change it, with a quick diagnostic to tell whether crawl demand is your real problem at all.
What crawl demand means
Google’s crawl budget is made of two parts. Crawl capacity is how much Googlebot can crawl without straining your server. Crawl demand is how much it wants to. In Google’s large-site crawl budget guide, demand varies with a site’s size, update frequency, page quality and relevance compared with other sites. The practical consequence: even if your server could handle ten times more requests, Googlebot will not use that headroom unless it has reasons to.
Google lists three factors behind demand:
- Perceived inventory. Without guidance, Google tries to crawl all or most URLs it knows about on your site. If it knows about 400,000 filter and session URLs, that inventory competes with your 5,000 real pages.
- Popularity. URLs that are more popular on the internet tend to be crawled more often to keep them fresh. Links, both external and internal, are the clearest popularity signal you control.
- Staleness. Google wants to recrawl documents often enough to pick up changes. Pages that change meaningfully and often are revisited more; pages that never change drift to longer intervals.
Google also notes that site-wide events such as a site move can trigger a temporary rise in demand while it reprocesses content under new URLs, which is why you often see crawl spikes after a migration.
Crawl demand vs crawl capacity vs crawl budget
| Term | Question it answers | Mainly controlled by | Typical symptom when it is the limit |
|---|---|---|---|
| Crawl capacity | How much can Googlebot crawl safely? | Server speed, errors, hosting | Crawling drops after slow responses or 5xx errors; host status warnings |
| Crawl demand | How much does Google want to crawl? | URL inventory, links, freshness, quality | Fast server but new pages wait weeks; Googlebot busy on parameter URLs |
| Crawl budget | What actually gets crawled | The lower of the two | “Discovered, currently not indexed” grows on large sites |
For more on the capacity side and when budget matters at all, see my guide to crawl budget. Google is clear that most sites do not need to worry: its guide is aimed at roughly 1 million+ page sites that change weekly, or 10,000+ page sites that change daily. A 200-page service site with slow indexing almost always has a quality or linking problem, not a crawl demand problem.
Is crawl demand really your problem?
Does your site have more than about 10,000 indexable URLs, or add hundreds of new URLs a week?
Does the Crawl Stats report show host status problems, rising response times or many 5xx responses?
In your logs or a crawl, are a large share of Googlebot requests going to parameter, filter, search or duplicate URLs?
Are the pages that wait longest weakly linked (deep in pagination, orphaned, or only in the sitemap)?
Result: probably not crawl demand. On smaller sites, slow indexing usually means Google does not see enough value in the pages. Check for thin or duplicate content, internal links and whether the page answers a real search need.
Result: crawl capacity first. Google slows down when your server struggles. Fix response times and errors before working on demand, or extra demand will not be served.
Result: inflated inventory. Demand is being spent on URLs you do not want. Control faceted navigation, block crawl traps, consolidate duplicates and return 404 or 410 for removed pages.
Result: weak popularity signals. Bring important pages closer to the homepage, link them from hubs and related pages, and earn external links to key sections.
Result: freshness and quality. Make genuine updates to key pages, send accurate lastmod dates, and review whether pages that are not crawled add anything new compared with similar pages.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
How to see your crawl demand
There is no “crawl demand” number in any tool. You infer it from what Googlebot does.
Search Console Crawl Stats
Open Settings, then Crawl stats. The Crawl Stats report shows total requests, download size, average response time and host status, plus breakdowns by response code, file type, Googlebot type and purpose. The purpose split is the most useful view for demand:
- Discovery requests are URLs Google has never crawled before. A healthy share means new content is being picked up.
- Refresh requests are recrawls of known URLs. Check which URLs are refreshed most; if it is your filter pages rather than your products, demand is misallocated.
Click into each breakdown to see example URLs. When I review this report, I look for patterns: a single parameter appearing again and again, old redirected URLs still being refreshed, or 404s for pages removed years ago.
Server logs
Logs show every Googlebot request, so you can count hits per section and per template over time. Log file analysis answers the questions Search Console samples cannot, such as “how many days on average between Googlebot visits to a product page?” Verify that requests really come from Google before trusting the numbers.
Indexing reports
In the Pages report, “Discovered, currently not indexed” means Google knows the URL but has not crawled it yet. A growing number on a large site is one of the clearest signs that demand (or capacity) is not keeping up. My guide to the Search Console coverage report explains each status.
How to influence crawl demand
1. Shrink the inventory Google has to think about
This is usually the biggest win. Every unnecessary URL Google knows about competes for attention.
- Stop filters, sort orders and internal search from creating unlimited crawlable URLs. My guide to faceted navigation covers which combinations to keep crawlable.
- Find and close crawl traps such as infinite calendars and session IDs.
- Return 404 or 410 for pages that are gone, rather than redirecting everything to the homepage.
- Merge near-duplicate and thin pages. Reducing index bloat lowers the inventory and concentrates signals on what remains.
2. Raise popularity of the pages that matter
Internal links are the popularity signal you fully control. Link new and important pages from the homepage, category hubs and related articles on the day they go live, rather than leaving them at the bottom of a paginated archive. External links to key sections help too; earning them is slower, but a link to a category page tends to lift crawling of everything beneath it.
3. Keep content genuinely fresh
Staleness works both ways. Pages that change in meaningful ways (new stock, new prices, updated advice) give Google a reason to come back. Changing a date or shuffling a sentence does not, and Google’s systems compare what actually changed. On blogs, I prefer to update a smaller set of important posts properly over touching hundreds lightly.
4. Send accurate sitemap signals
Keep your XML sitemap limited to canonical, indexable URLs. Google’s sitemap documentation says it uses the lastmod value if it is consistently and verifiably accurate, and ignores priority and changefreq. A sitemap that marks every URL as modified today teaches Google to ignore your lastmod values.
5. Support efficient recrawling
Return a 304 Not Modified response when a page has not changed since Google last crawled it, if your server supports conditional requests. It saves resources on both sides, which matters most on large sites.
Worked example: a 6,000-product shop with 300,000 crawled URLs
This is an illustrative example, not a client. A clothing shop has about 6,000 products and 150 categories. Logs show Googlebot requesting around 300,000 distinct URLs over a month, most of them combinations like ?colour=red&size=m&sort=price. New-season products take weeks to appear in search.
- Diagnosis: server responses are fast and host status is clean, so capacity is fine. The problem is inventory: demand is being spread across filter URLs.
- Fix 1: keep a small set of useful filter pages (for example “red lawn suits”) as static, linked, indexable URLs; make the remaining filter combinations non-crawlable links and block their parameter patterns in robots.txt.
- Fix 2: link the new-season category from the homepage and main menu, and list new arrivals on each relevant category’s first page.
- Fix 3: clean the sitemap to products and categories only, with lastmod set when price, stock or content actually changes.
- What to watch: in Crawl Stats and logs, the share of Googlebot requests going to product and category URLs, and the time from publishing to first crawl for new products.
Mistakes that waste crawl demand
- Using noindex to save crawl. Google still has to crawl a noindexed page to see the tag. If you want Google to stop requesting URLs, stop linking them or block them in robots.txt.
- Fake freshness. Updating dates without changing content, or regenerating the sitemap with today’s date for every URL.
- Redirect chains after migrations. Every hop is another request. Update internal links to point at final URLs, as covered in my migration checklist.
- Soft 404s. Empty category and out-of-stock pages that return 200 keep getting recrawled. Return proper status codes or add useful content.
- Chasing crawl numbers. More Googlebot requests is not a ranking factor in itself. The goal is the right pages crawled promptly.
Check your understanding
Your server is fast and shows no errors, but new pages wait weeks to be crawled. What is most likely limiting crawling?
- Crawl capacity
- Crawl demand
- The robots.txt crawl-delay directive
With a healthy server, capacity is not the constraint. Google simply does not see enough reason to crawl those URLs sooner, which is demand.
Which sitemap field does Google say it uses when accurate?
- priority
- changefreq
- lastmod
Google uses lastmod when it is consistently and verifiably accurate, and ignores priority and changefreq.
Which change most directly reduces “perceived inventory”?
- Stopping filters from generating crawlable URLs
- Adding more images to product pages
- Submitting the sitemap again every day
Perceived inventory is the set of URLs Google knows about. Removing crawl paths to endless filter combinations shrinks it.
Does adding noindex to thousands of filter pages stop Google crawling them?
- Yes, immediately
- No, Google must crawl a page to see noindex
- Only if the sitemap lists them
Noindex controls indexing, not crawling. Google may crawl noindexed pages less over time, but to stop requests you need to remove links or block crawling.
Frequently asked questions
What is crawl demand in SEO?
Crawl demand is how much Google wants to crawl a site’s URLs. Google bases it on the URLs it knows about, how popular they are and how stale its stored copies are. Together with crawl capacity, it determines your crawl budget.
What is the difference between crawl demand and crawl budget?
Crawl budget is the set of URLs Googlebot can and wants to crawl. Crawl demand is the “wants to” half; crawl capacity is the “can” half, based on your server’s health.
How can I increase crawl demand?
Remove or block low-value URLs, link important pages prominently, earn external links to key sections, update content in meaningful ways and keep sitemap lastmod dates accurate. You cannot request a higher crawl rate directly.
Does crawl demand affect rankings?
Not directly. Crawling is a prerequisite for indexing, so low demand can delay new or updated pages reaching search results. Being crawled more often does not make a page rank higher.
Do small websites need to worry about crawl demand?
Rarely. Google’s guidance targets large or fast-changing sites. On small sites, slow indexing usually points to content quality, duplication or weak internal links.
Why did my crawl requests suddenly increase?
Common causes are a site move, a large batch of new URLs, a new parameter or filter creating URLs, or a section gaining popularity. Check Crawl Stats by purpose and file type to see which URLs are behind the spike.
Sources
- Google Search Central, “Crawl budget management for large sites” (accessed 2026).
- Google Search Central, “Build and submit a sitemap” (accessed 2026).
- Google Search Console Help, “Crawl Stats report” (accessed 2026).
If you run a large site and Googlebot seems to be spending its time in the wrong places, my technical SEO services include crawl and log analysis, or you can ask me to take a look.
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.