Short answer: Log file analysis for SEO means reading your server’s raw access logs to see which URLs Googlebot, Bingbot and AI crawlers actually request, how often, and what status code they get back. You collect 30 to 90 days of logs, parse each line into fields, verify the bots are genuine, then group requests by site section and status code to find wasted crawling, errors, orphan pages and important pages bots ignore.
This guide is for SEOs, developers and site owners who suspect crawling is the reason pages are slow to index, and who want to read their own server logs rather than guess. By the end you will be able to get the files, parse them with a few commands or a tool, verify which bots are real, spot the patterns that matter and turn them into fixes. I use exactly this process on technical audits, migrations and large e-commerce sites.
What a server log contains
Every time Googlebot, Bingbot, GPTBot or any other client requests a URL, your server writes one line to its access log. Crawling tools simulate what a bot might do and Search Console summarises what Google chooses to report, but the log records what actually happened, request by request. Most servers (Apache, Nginx, LiteSpeed, IIS) use Common Log Format, Combined Log Format or W3C Extended. A typical Combined Log Format line looks like this:
66.249.66.1 - - [14/Oct/2026:09:12:44 +0000] "GET /services/technical-seo/ HTTP/1.1" 200 18432 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
| Field | Value in the example | Why it matters for SEO |
|---|---|---|
| IP address | 66.249.66.1 | Needed to verify that a “Googlebot” is really Google |
| Timestamp | 14/Oct/2026:09:12:44 +0000 | Crawl frequency, trends and time to first crawl |
| Method and URL | GET /services/technical-seo/ | Which pages and parameters bots spend time on |
| Protocol | HTTP/1.1 | Context only; Googlebot can also crawl over HTTP/2 |
| Status code | 200 | Errors, redirects and soft 404s bots receive |
| Bytes sent | 18432 | Heavy pages and resources |
| Referrer | – | Usually empty for bots; useful for humans |
| User agent | Googlebot Smartphone string | Which bot claims to be visiting |
Some servers and CDNs can also log response time, cache status and host name. If you can switch on response time, do it. It is one of the most useful fields for SEO and it is usually off by default.
Why log file analysis matters for SEO
Search Console’s Crawl Stats report (under Settings) is a good starting point, but Google’s help page is clear that it covers Google’s crawling only, that its example URLs are a representative sample, and that its numbers can differ from your server logs. Logs give you every request from every bot, which lets you answer questions no other source can:
- Is crawling being wasted? Faceted navigation, sort and filter parameters, session IDs, internal search and calendar archives can absorb a large share of bot requests. I explain the concept in what crawl budget is and how to optimise it.
- Are important pages crawled at all? New products, updated service pages and key articles that bots rarely request will be slow to index and slow to reflect changes.
- What errors do bots actually get? Intermittent 5xx errors and timeouts often appear in logs before Search Console reports them. Google says that when a site returns server errors or rate-limiting signals such as 429, its crawl capacity limit goes down and it crawls less (Google Search Central).
- Are redirects respected? If bots keep requesting old URLs months after a migration, internal links or sitemaps still point at them.
- Which AI crawlers visit, and how much? Logs show whether GPTBot, ClaudeBot, PerplexityBot and others reach your content, and whether your robots.txt rules work.
- Do bots find orphan pages? URLs bots request that your site no longer links to usually come from old sitemaps, external links or leftover templates.
Do small sites need it?
Google’s own guidance on crawl budget is aimed at sites with roughly a million or more pages that change weekly, or 10,000 or more that change daily, and its Crawl Stats help says sites under about a thousand pages usually do not need that report. My rule follows the same logic: for a well-indexed site of a few hundred pages, logs are a check you run when something looks wrong. For e-commerce, marketplaces, publishers and sites with tens of thousands of URLs, logs are often where the real problem is found.
Not sure if this is hurting your site?
Get the fixes that matter most, in priority order.
How to get your log files
- VPS or dedicated server: usually
/var/log/nginx/access.logor/var/log/apache2/access.log(/var/log/httpd/on some distributions), with older files rotated and compressed as.gz. - Shared hosting and cPanel: look for “Raw Access” in cPanel. Many Pakistani businesses I work with are on shared cPanel plans where raw logs are kept for only a few days, so turn on log archiving or download the files weekly before you need them.
- Managed WordPress hosts: some provide access logs in the dashboard or over SFTP; others only through support.
- CDNs and load balancers: if Cloudflare, Fastly, Akamai or a cloud load balancer sits in front of the site, many bot requests are answered at the edge and never reach your origin. You need the CDN’s logs too (for example Cloudflare Logpush or AWS CloudFront logs), or you will undercount crawling.
Aim for at least 30 days of data, and 60 to 90 days for large sites or around a site migration. If the site runs on several servers, collect from all of them.
Privacy: IP addresses are personal data under GDPR, UK GDPR and similar laws. Filter to bot traffic early, keep raw files secure, and strip or anonymise human IPs before sharing logs with an agency or consultant. When clients send me logs, I ask for bot lines only where their setup allows it.
How to parse server logs
Parsing turns each raw line into columns you can filter and count. There are three practical routes, and most audits use more than one.
1. Command line for a quick look
On Linux or Mac, a few commands answer basic questions in seconds. To count Googlebot requests by status code:
grep "Googlebot" access.log | awk '{print $9}' | sort | uniq -c | sort -rn
To list the URLs Googlebot requests most often:
grep "Googlebot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -50
To see how many Googlebot requests hit URLs with a query string:
grep "Googlebot" access.log | awk '{print $7}' | grep -c "?"
Field positions vary with the log format, so check a sample line first. Use zcat or zgrep for compressed files.
Worked example: reading the output of the status code command
Suppose the first command returns 8,400 lines with 200, 1,900 with 301, 600 with 404 and 100 with 503, out of 11,000 Googlebot requests in 30 days. That means about 17% of Googlebot’s requests went to redirects and about 5% to missing pages. Neither number is a disaster on its own, but the next step is clear: run the second command filtered to 301s (grep "Googlebot" access.log | awk '$9==301 {print $7}' | sort | uniq -c | sort -rn | head) to see which old URLs keep being requested, then find the internal links or sitemap entries that still point to them. The 503s need a look at the timestamps, because errors clustered at the same hour usually mean a backup job or traffic spike is overloading the server. These figures are illustrative, not from a client.
2. Dedicated log analysis tools
- Screaming Frog Log File Analyser: a desktop tool that imports common formats, verifies search bots and combines log data with a crawl to find orphan and uncrawled URLs. The free version has a small event limit.
- Semrush Log File Analyzer: browser-based, fine for a first look at bot activity and status codes.
- JetOctopus, Botify, Oncrawl and similar: cloud platforms built for large sites, with continuous log ingestion and crawl comparisons.
3. Spreadsheets, Python or a data warehouse
For small sites, a filtered export into Excel or Google Sheets with pivot tables on status code, URL and user agent is enough. For millions of lines, Python with pandas, BigQuery or an ELK stack scales far better, and scripts can run weekly so you see trends rather than one snapshot. Whatever the method, standardise the format first: custom log formats are the most common reason imports fail.
Verify the bots before you trust the data
User agents can be faked, and scrapers often pretend to be Googlebot. Unverified data leads to wrong conclusions, so verify before you count anything.
- Google: Google’s verification guide says to run a reverse DNS lookup on the IP, confirm the host ends in
googlebot.com,google.comorgoogleusercontent.com, then run a forward lookup on that host and check it returns the same IP. For bulk checks, match IPs against Google’s published JSON lists such ascommon-crawlers.jsonandspecial-crawlers.json. - Bingbot: reverse DNS should resolve to
search.msn.com, and Bing Webmaster Tools has a verification tool. - AI crawlers: OpenAI and Perplexity publish IP ranges for their bots. Where an operator publishes ranges, check against them; where it does not, treat high-volume requests with that user agent as unverified.
For a one-off check, host 66.249.66.1 on the command line, or my free DNS lookup tool, gives you the reverse record. Also separate Google’s crawler types. Googlebot Smartphone should account for most crawling on a mobile-first site; Googlebot Desktop, Googlebot-Image, Googlebot-Video, Storebot-Google, AdsBot-Google, Google-InspectionTool (used by URL Inspection) and GoogleOther each have different jobs and should not be counted as one bot.
AI crawlers to look for in your logs
AI companies now crawl both to train models and to answer users’ questions in real time. Their behaviour can differ a lot from Googlebot’s, so I segment them separately on every audit.
| User agent | Operator | What it is for |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train OpenAI’s models |
| OAI-SearchBot | OpenAI | Finding pages to show in ChatGPT search results |
| ChatGPT-User | OpenAI | Fetching a page when a ChatGPT user’s request needs it |
| ClaudeBot | Anthropic | Collecting content that may be used for model training |
| Claude-SearchBot and Claude-User | Anthropic | Search indexing and user-requested fetches for Claude |
| PerplexityBot | Perplexity | Indexing pages to surface and link in Perplexity answers |
| Perplexity-User | Perplexity | Fetching pages in response to a user’s question |
| CCBot | Common Crawl | Open web crawl widely used as AI training data |
| Bytespider | ByteDance | Crawling associated with ByteDance products and AI models |
You will not find Google-Extended in your logs. Google’s crawler documentation says Google-Extended has no separate user agent string; it is a robots.txt control token, and it does not affect inclusion or ranking in Google Search. Google crawls with its normal Googlebot user agents, so any Gemini-related fetching simply appears as Googlebot.
When you review AI crawlers, check:
- Which bots visit, how often and which sections they request.
- Whether bots you have disallowed still appear and get 200 responses. If they do, the rule is wrong or the bot ignores it, and a CDN or firewall rule is the next step. My robots.txt tester and AI crawler checker shows what your current file allows.
- Whether bots you want, such as OAI-SearchBot or PerplexityBot, are being blocked by a security plugin and receiving 403 errors. I find this more often than deliberate blocking.
- Server load from aggressive crawling, especially on small hosting plans.
Whether to allow or block each one is a business decision. I explain the trade-offs and the syntax in AI crawlers explained, and the basics of the file in my guide to robots.txt.
What to look for: from findings to fixes
Once logs are parsed and bots verified, segment requests by bot, by site section (products, categories, blog, parameters, assets) and by status code. These are the patterns I check on every audit:
| What you see in the logs | What it usually means | Typical fix |
|---|---|---|
| High share of bot hits on parameter, filter or sort URLs | Faceted navigation creating a near-infinite URL space | Stop linking to low-value combinations, use canonicals, disallow clear crawl traps |
| Key pages crawled rarely or not at all | Weak internal linking, deep click depth or missing from sitemaps | Link from hubs and navigation; add to XML sitemaps with accurate lastmod |
| Many requests returning 301 or 302 | Internal links and sitemaps still pointing at old URLs, or redirect chains | Update links to final URLs and collapse chains to one hop |
| Repeated 404 or 410 requests | Deleted pages still linked internally, in sitemaps or from other sites | Remove internal links; redirect where a relevant replacement exists |
| 5xx errors or timeouts at certain times | Server capacity, slow queries, backups or plugin problems | Fix with hosting or developers; Google crawls less when it sees server errors |
| 200 responses on thin or empty pages | Soft 404s, such as empty category or search pages | Return 404 or 410, or noindex and remove links |
| Bots requesting URLs not found in your site crawl | Orphan pages, old sitemaps or leftover URLs | Link valuable orphans; redirect or remove the rest |
| Disallowed AI bots still getting 200 responses | Robots.txt mistake or a bot ignoring robots.txt | Correct the rule; enforce at the CDN or firewall if needed |
Two more checks are worth the time. First, compare Googlebot Smartphone to Desktop requests; if desktop dominates, check whether the mobile version serves the same content. Second, measure how many days pass between publishing or updating a key URL and its first Googlebot request. That tells you more about crawl priority than any total. My posts on crawl traps, crawl demand and faceted navigation go deeper. To confirm what a flagged URL returns right now, run it through my redirect and status code checker.
Crawl waste calculator
These bands are my working thresholds from audits, not Google figures. Some parameter URLs are legitimate (for example paginated categories), so check the patterns before blocking anything. Overlap between the two numbers is normal.
How to run a log-based SEO audit
This is the order I work in. Tick each step as you go.
Log file audit checklist
- Set the question: write down what you want to learn (wasted crawling, uncrawled pages, migration checks, AI bots) before opening a file.
- Collect the logs: 30 to 90 days from every origin server and the CDN.
- Clean and parse: standardise the format, split into fields, remove uptime monitors and your own tools.
- Strip personal data: keep bot lines, drop or anonymise human IPs.
- Verify bots: confirm Googlebot, Bingbot and AI crawlers by DNS or published IP ranges and set aside fakes.
- Split Google’s crawlers: Smartphone, Desktop, Image, AdsBot and GoogleOther counted separately.
- Crawl the site: run Screaming Frog or Sitebulb and export your XML sitemap URLs.
- Match the datasets: find URLs crawled but not linked (orphans) and URLs linked but never crawled.
- Segment and count: requests by bot, site section, status code and day.
- Cross-check Search Console: compare with Crawl Stats and the page indexing report.
- Prioritise fixes: rank by how many important URLs each issue affects and how easy it is to fix.
- Re-check: analyse fresh logs four to eight weeks after the fixes.
Ticks are saved in your browser.
The cross-check step is where logs pay off. If an important page shows as “Discovered, currently not indexed” in Search Console and the logs show Googlebot never requested it, the problem is crawl priority and internal linking, not the content itself. My guide to the Search Console coverage report explains those statuses, and keeping your XML sitemap accurate helps bots find the right URLs.
On a mid-sized site, the first full pass usually takes about a working day once the logs are in hand; getting access to the logs often takes longer than the analysis. Run one after a migration or redesign, when indexing drops without an obvious cause, and before and after large navigation changes. Monthly suits big e-commerce and publishing sites; quarterly is enough for most others.
Common mistakes in log file analysis
- Not verifying bots, so fake Googlebots inflate crawl numbers and send you chasing the wrong problem.
- Ignoring CDN logs, which can hide most crawling on a cached site.
- Analysing one day, which shows noise rather than patterns.
- Treating every Google user agent as Googlebot, mixing AdsBot and Googlebot-Image into the totals.
- Reading logs in isolation, without a crawl, sitemap or Search Console data for context.
- Blocking without checking, such as disallowing a parameter that also appears on canonical URLs.
- Keeping raw IPs longer than needed, which adds privacy risk without SEO value.
Log analysis sits inside the wider process of how search engines crawl and index pages. It is part of my technical SEO services, and for online shops with faceted navigation it is often where my e-commerce SEO work begins.
Frequently asked questions
What is log file analysis in SEO?
It is the review of your server’s access logs to see exactly how search engine and AI crawlers request your pages: which URLs, how often and with what status codes. It reveals crawl waste, errors and ignored pages that crawling tools and Search Console can miss.
What is the difference between log file parsing and log file analysis?
Parsing is the mechanical step of splitting each raw line into fields such as IP, timestamp, URL, status code and user agent. Analysis is what you do with the parsed data: filtering to verified bots, grouping by section and status, and deciding what to fix.
How is log data different from Google Search Console?
The Crawl Stats report covers Google only, shows aggregated totals and a sample of example URLs, and can differ from your logs. Logs record every request from every bot, including Bing and AI crawlers, but need parsing and verification first.
Do small websites need log file analysis?
Usually not as a routine task. If a small site is indexed well, Search Console covers most needs. Logs become useful when there is an unexplained indexing problem, after a migration, or when you want to see how AI crawlers treat your content.
Why don’t I see Google-Extended in my logs?
Because it is a robots.txt token, not a crawler. Google fetches pages with its normal user agents, and Google-Extended only controls whether that content may be used for Gemini. It does not affect rankings in Search.
How long should I keep log files for SEO?
Keep at least 30 days of bot-filtered logs, and 90 days for large sites so you can compare before and after changes. Keep human IP data only as long as you genuinely need it.
Sources
- Cloudflare, “From Googlebot to GPTBot: who’s crawling your site in 2025” (July 2025).
- Google Search Central, “Large site owner’s guide to managing your crawl budget” (updated 2026).
- Google Search Central, “Verifying Googlebot and other Google crawlers” (updated 2026).
- Google Search Central, “Google’s common crawlers” (accessed 2026).
- Search Console Help, “Crawl Stats report” (accessed 2026).
If you would like a second opinion on whether a log review is worth doing for your site, my SEO audit service includes one where the site is large enough to need it, or you can send me a question.
Request received
Thank you! Saad will contact you within the next 24 hours.
Keep an eye on your phone and inbox.