Short answer: Crawling in SEO is the stage where search engine bots, such as Googlebot, request the URLs they know about, download the content and follow links to find more pages. It comes before indexing (storing and understanding the page) and ranking (choosing it for a query). If Google cannot crawl a page, it usually cannot index or rank its content, so crawl access is the first thing to check when a page is missing from search.
This guide is for site owners, writers and new SEOs who want to understand how search engines find pages, and why some pages never show up. By the end you will be able to explain crawling in plain terms, tell a crawling problem apart from an indexing or ranking problem, and run the checks I use when a client tells me “Google can’t see my page”.
What is crawling in SEO?
Crawling is how a search engine finds out what exists on the web. A program called a crawler (also called a bot or spider) takes a list of URLs, requests each one from the server, reads what comes back and adds any new links it finds to the list. Google’s crawler is Googlebot; Bing’s is Bingbot. AI companies now run their own crawlers too, which I cover in my guide to AI crawlers and robots.txt.
Google describes Search as working in three stages, and its How Search Works documentation is clear that “not all pages make it through each stage”. The first stage, which Google calls URL discovery, is simply finding out what pages exist. Google also states that it does not guarantee it will crawl, index or serve any page, which is worth remembering before you blame an update for a missing URL.
Crawling, indexing and ranking: the difference
These three words get mixed up constantly, and mixing them up leads to the wrong fix. Here is how I separate them when diagnosing a problem:
| Stage | What happens | Typical problem | Where to check |
|---|---|---|---|
| Crawling | Googlebot discovers the URL and downloads it | Blocked by robots.txt, server errors, no links pointing to the page | URL Inspection, Crawl Stats, server logs |
| Indexing | Google renders the page, works out what it is about and decides whether to store it | noindex tag, duplicate of another URL, thin or low-value content | Page indexing report, URL Inspection |
| Ranking | Google picks indexed pages to show for a query and orders them | Weak relevance, poor quality, stronger competitors | Search Console Performance report |
A useful rule: if URL Inspection says “URL is not on Google” and the last crawl is missing or failed, you have a crawling problem. If it was crawled but not indexed, it is an indexing problem, which my guide to the search engine indexing process covers in detail. If it is indexed but gets no impressions, work on content and relevance, not crawl settings.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
How Google crawls a page, step by step
- Discovery. Google finds URLs from links on pages it already knows, from XML sitemaps you submit, and from URLs you ask it to inspect. A page with no internal links and no sitemap entry (an orphan page) may never be found.
- Checking robots.txt. Before fetching URLs on a host, Googlebot reads its robots.txt file to see which paths it is allowed to request.
- Fetching. Googlebot requests the URL. Most requests now come from Googlebot Smartphone, because Google primarily indexes the mobile version of content, according to Google’s Googlebot documentation. The same page notes that for most sites Googlebot should not visit more than once every few seconds on average.
- Reading the response. The HTTP status code decides what happens next: a 200 is processed, a 301 sends Googlebot to the new address, a 404 tells it the page is gone, and 5xx errors tell it the server is struggling, so it slows down.
- Rendering. Google runs the page’s JavaScript using a recent version of Chrome, so content added by scripts can be seen, but only if the scripts and files it needs are not blocked. My guide to rendering in SEO explains where this goes wrong.
- Scheduling a revisit. Google uses an algorithmic process to decide how often to come back. Pages that change often and matter to users tend to be recrawled more often.
Crawl budget: does it matter for your site?
Crawl budget is the number of URLs Google can and wants to crawl on your site in a given period. Google’s crawl budget guide splits it into two parts: the crawl capacity limit (how much connection time your server can give Google without slowing down) and crawl demand (how much Google wants to crawl, based on popularity and freshness). Crawl demand is the half you influence most, through useful content and clear internal links.
The honest answer for most sites: you do not need to worry. Google’s guide is written for sites with roughly a million or more unique pages, or 10,000+ pages that change daily, and it says smaller sites mainly need an up-to-date sitemap and a regular look at the Page indexing report. Where I do see crawl waste on smaller sites is when a plugin or filter system creates thousands of junk URLs, such as calendar pages that go on forever or every combination of product filters. That is a crawl trap, and fixing it is worth doing at any site size. For the bigger picture, see my full guide to crawl budget.
What helps and what blocks crawling
Robots.txt
The robots.txt file tells well-behaved crawlers which paths they may request. The protocol is now an official internet standard, RFC 9309, published in September 2022, and it states plainly that its rules “are not a form of access authorization”. Two practical consequences: robots.txt will not keep private pages secret (use a login), and blocking a URL in robots.txt does not remove it from Google, because Google cannot crawl the page to see a noindex tag on it. Test changes with my free robots.txt tester before you publish them.
XML sitemaps
A sitemap lists the URLs you want crawled, with optional last-modified dates. It helps discovery, especially for new sites and deep pages, but it does not force Google to crawl or index anything. Keep it to canonical, indexable URLs that return 200. My guide to XML sitemaps shows how to build and submit one.
Internal links and site structure
Links are still the main way Googlebot moves around your site. Pages linked from the main navigation and from related articles get found and refreshed faster than pages buried five or six clicks deep. Use normal <a href> links; links that only work through JavaScript click handlers may not be followed.
Server health and status codes
If your server returns errors or responds slowly, Google reduces how fast it crawls. Redirect chains (A to B to C) waste requests, and soft 404s (an error page that returns 200) confuse both crawling and indexing.
Common crawl problems I find in audits
- A leftover
Disallow: /from a staging site that went live with the developer’s robots.txt still in place. - A security plugin or firewall blocking Googlebot. On shared hosting, which many Pakistani small business sites use, I regularly see bot protection rules that challenge or block crawlers, so Search Console shows server errors while the site looks fine in a browser.
- CSS and JavaScript blocked in robots.txt, so Google renders a broken page.
- Orphan pages that exist in the CMS but are linked from nowhere.
- Redirect chains after a migration, where old URLs pass through two or three hops. My free redirect checker shows every hop.
- Endless parameter URLs from sorting, filters or session IDs.
- Very heavy HTML pages with inline code, where important content sits beyond Googlebot’s 2MB fetch limit.
How to check whether Google is crawling your site
- URL Inspection in Search Console. Enter the URL. It shows whether the page is on Google, when it was last crawled, whether crawling was allowed and which user agent crawled it. Use “Test live URL” to see what Googlebot gets right now.
- Crawl Stats report. Found under Settings, it shows Google’s crawling history, response codes, file types and whether each request was discovery (a new URL) or refresh (a known page). Google’s Crawl Stats help page says host status covers the past 90 days and that sites with fewer than a thousand pages should not normally need the report, so use it when something looks wrong, not as a weekly habit.
- Page indexing report. Look at “Discovered, currently not indexed” (Google knows the URL but has not crawled it yet) and “Blocked by robots.txt”. My guide to the Search Console coverage report explains each status.
- Server logs. For large sites, logs show exactly which URLs Googlebot requested and how often. See log file analysis for the method.
Why isn’t Google crawling my page?
Does URL Inspection’s live test say crawling is allowed?
Does the live test fetch the page successfully with a 200 status?
Is the page linked from at least one other crawlable page and listed in your sitemap?
Has it been more than two weeks since you published or requested indexing?
Result: blocked by robots.txt. Find the Disallow rule that matches the URL, remove or narrow it, test with a robots.txt tester, then request indexing.
Result: server or response problem. Check for 5xx errors, firewall or bot protection rules, redirects that loop, or a soft 404. Fix the response first; Google will not crawl reliably until it does.
Result: discovery problem. Add internal links from relevant, already-indexed pages and include the URL in your XML sitemap.
Result: give it time. New pages on small sites are often crawled within days, but not always. Request indexing once and wait; repeated requests do not speed it up.
Result: probably a quality or priority signal. Google can crawl it but has chosen not to yet. Improve the content, strengthen internal links from important pages, and check whether a near-duplicate page already covers it.
How to make your site easier to crawl
- Keep important pages within three clicks of the homepage and link to them from related content.
- Submit one clean XML sitemap in Search Console and keep it current.
- Allow CSS, JavaScript and image files that pages need to render.
- Return correct status codes: 404 or 410 for removed pages, 301 for moved ones, never 200 for errors.
- Use canonical tags on duplicates and keep parameter URLs out of internal links.
- Fix server errors quickly and make sure hosting can cope with bots as well as visitors.
- Serve important content in the initial HTML where you can, rather than relying only on JavaScript.
Worked example: a robots.txt that blocks too much
A WordPress site had these lines: Disallow: /wp-content/ and Disallow: /*?. The first blocked the theme’s CSS and JavaScript, so Google rendered an unstyled, half-empty page. The second blocked every URL with a parameter, including the paginated blog archive that linked to older posts. The fix was to remove the /wp-content/ rule entirely (keeping only Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php) and replace the parameter rule with specific rules for the sort and filter parameters that created duplicates. The live test in URL Inspection then showed the full page.
Check your understanding
A page is blocked in robots.txt and also has a noindex tag. What happens?
- Google reads the noindex and drops the page
- Google cannot crawl the page, so it never sees the noindex
- Both rules cancel out and the page ranks normally
Robots.txt stops the fetch, so the noindex tag is never read. To remove a page from Google, allow crawling and use noindex.
Your 200-page site has a few pages not indexed. What is the most likely cause?
- Crawl budget has run out
- Googlebot only crawls sites once a year
- Weak internal linking, duplication or low-value content
Google says crawl budget mainly matters for very large or fast-changing sites. On small sites the cause is usually discovery or quality.
Which crawler does most of Google’s crawling today?
- Googlebot Smartphone
- Googlebot Desktop
- Googlebot Image
Google primarily indexes the mobile version of content, so most requests use the mobile crawler.
Frequently asked questions
What is the difference between crawling and indexing?
Crawling is Googlebot finding and downloading a page. Indexing is Google processing that page, understanding it and deciding whether to store it in its index. A page must normally be crawled before it can be indexed, but a crawled page is not guaranteed to be indexed.
How often does Google crawl a website?
There is no fixed schedule. Google decides algorithmically, based on how often pages change, how important they seem and how well the server responds. Busy news sites may be crawled constantly; a small brochure site may see some pages revisited only every few weeks.
How do I get Google to crawl my site faster?
Link to new pages from pages Google already visits often, keep your sitemap updated, fix server errors and slow responses, and use URL Inspection to request indexing for important new URLs. There is no setting that forces faster crawling.
Can I stop Google from crawling a page?
Yes, with a Disallow rule in robots.txt. But that only stops crawling. If you want a page out of search results, use a noindex tag and leave it crawlable, or put it behind a login.
Do AI crawlers follow robots.txt?
The major ones, such as OpenAI’s GPTBot and Google-Extended (a control token rather than a separate crawler), publish their user agents and say they respect robots.txt. Blocking them has no effect on Googlebot’s crawling for Search, so decide on each one separately.
Sources
- Google Search Central, “In-depth guide to how Google Search works” (last updated December 2025).
- Google Search Central, “Googlebot” (last updated February 2026).
- Google Search Central, “Crawl budget management for large sites” (accessed 2026).
- Search Console Help, “Crawl Stats report” (accessed 2026).
- IETF, “RFC 9309: Robots Exclusion Protocol” (September 2022).
If pages on your site are not being crawled or indexed and you cannot see why, my technical SEO services start with exactly these checks, or you can ask me a question directly.
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.