Home/Technical SEO / Crawling / Crawlability Audit
Crawlability Audit: How to Find and Fix Technical SEO Issues
A crawlability audit is the process of confirming that search engine and AI crawlers can fetch, discover, and follow links through the pages you want indexed — and pinpointing exactly where that access breaks down when it doesn’t. This is the step-by-step audit process. If you need the underlying concepts first, start with SEORAF’s crawlability overview.
This article builds on the fundamentals in SEORAF’s Crawling hub and turns them into a complete, practical audit methodology — not a definitions glossary. It walks through defining scope, collecting evidence from multiple independent sources, checking each access layer in order, diagnosing conflicting signals, fixing what’s broken, and confirming the fix worked.
A Diagnostic Sequence for Telling Crawlability and Indexability Apart
Crawlability is whether a bot can fetch a URL at all. Indexability is whether a successfully fetched page is eligible to be indexed. A page can pass one gate and fail the other — the crawlability overview covers why that distinction exists. When a page isn’t performing in search, work through these questions in order, stopping as soon as one applies:
- Can the page be fetched at all? Test it directly (Step 3 below). If not, this is a crawlability problem at the access layer — nothing downstream matters until it’s fixed.
- Is it blocked in robots.txt? Also a crawlability problem, just a pre-fetch mechanism.
- Does it fetch cleanly but carry a noindex directive (meta robots or X-Robots-Tag)? This is an indexability problem — the page was crawled, then excluded afterward.
- None of the above, and still missing from the index? The cause likely sits outside crawlability — content quality, canonicalization, or crawl recency.
Step 1: Define the Audit Scope
Before collecting anything, fix what “the site” means for this audit — skipping this step is the most common reason an audit produces confusing findings later.
- Boundary: subdomains included, staging/dev excluded, non-HTML resources in or out of scope
- Objective: general health check, response to a traffic/indexing drop, pre-launch audit, or recurring scheduled audit
- Depth: full-site vs. a scoped section or template type
Step 2: Collect URLs From Multiple Sources
Crawlability can’t be evaluated against a single “true” URL list — evaluate it against several independently-sourced lists, because disagreement between the lists is itself diagnostic data.
| Source | What it represents |
|---|---|
| Sitemap-declared URLs | What the site claims should be crawled/indexed |
| Crawler-discovered URLs | What a crawl simulation finds by following links outward |
| Search-engine-reported URLs | What Google Search Console shows as known to Google |
| CMS/database export | What the site’s own system believes exists |
| Log-file-observed URLs | What bots have actually requested |
What This Audit Checks, and In What Order
| Signal | Checked in |
|---|---|
| robots.txt access | Step 3 |
| Meta robots / X-Robots-Tag | Step 4 |
| HTTP status codes & redirects | Step 5 |
| Internal linking & discovery | Step 6 |
| XML sitemap accuracy | Step 7 |
| Canonical conflicts | Step 8 |
| JavaScript rendering | Step 9 |
| Crawl traps & crawl budget | Step 10 |
| Server & access failures | Step 11 |
| Log-file triangulation | Advanced Diagnosis |
| AI crawler access | Scoped extension |
Step 3: Test robots.txt Access
- File is reachable at the root path and returns a successful response
- Every
Disallow/Allowrule mapped against real URL paths, not assumed from intent - Rules checked per user-agent block separately, per the Robots Exclusion Protocol (RFC 9309)
Sitemap:directive, if present, points to a currently valid sitemap
Illustrative example (not a real audited site): a rule intended to block a staging subdirectory, written as Disallow: /staging, also matches /staging-assets/ and any other path beginning with that string — easy to introduce, easy to miss without testing the rule directly.
robots.txt permissions aren’t universal — a rule allowing Googlebot doesn’t automatically apply to Bingbot, GPTBot, or ClaudeBot; each user-agent block is evaluated independently. For robots.txt syntax and directive reference, see SEORAF’s robots.txt guide.
Step 4: Test Meta Robots and X-Robots-Tag
robots.txt operates before a fetch. Meta robots and X-Robots-Tag operate after a successful fetch.
| Directive | Where it lives | How to check it |
|---|---|---|
<meta name="robots"> | Page <head> | View page source or rendered DOM |
X-Robots-Tag | HTTP response header | Inspect HTTP headers directly — not visible in page source |
X-Robots-Tag is easy to miss precisely because it’s invisible in page source — it’s commonly set at the server, CDN, or reverse-proxy level, sometimes left over from a staging configuration never removed after launch.
Step 5: Audit HTTP Status Codes and Redirects
| Status range | Audit action |
|---|---|
| 2xx | Confirm this is the expected outcome for the URL’s intended state |
| 3xx | Follow to final destination; check for chains and loops |
| 4xx | Confirm intentional; flag if internally linked, sitemap-listed, or previously indexed |
| 5xx | Escalate as an infrastructure issue — blocks crawling outright |
Soft 404s: a 200 OK status on a page that’s actually empty or functions as an error page. Google’s own guidance notes this pattern is confusing for both users and crawlers and can limit overall crawl coverage. Source: Google Search Central — How Google handles soft 404 errors.
Redirect chains and loops: a single redirect (A → B) is low-risk. A chain (A → B → C) adds fetch overhead. A loop — eventually pointing back to a previously-visited URL — is a harder failure: the destination is never reached.
Illustrative example: an old product URL redirects to a category page later redirected to a new structure, producing a two-hop chain where a direct redirect to the current destination would suffice.
Step 6: Audit Internal Linking and Discovery
An orphan page — no internal links pointing to it — is effectively invisible to a crawler relying on link-following for discovery.
- Confirm links use real, crawlable
<a href>markup, not JavaScript-only click handlers - Check click depth from key entry points
- Cross-reference against Step 2’s crawler-discovered list: a URL in the CMS but not there is a strong orphan candidate
- Confirm internal links point to live, canonical versions — not redirected or non-canonical duplicates
An orphan page (zero internal links anywhere) and a deep page (linked, but many hops from entry points) are different problems with different fixes. For discovery mechanics beyond this audit-specific check, see SEORAF’s URL discovery guide.
Step 7: Validate XML Sitemap Accuracy
- Sitemap reachable and well-formed (or split via a sitemap index for large sites)
- Every listed URL is canonical, indexable, and returns 200 — no redirects, noindexed URLs, or error pages
- No important, indexable URLs missing entirely
- Sitemap referenced via robots.txt and/or submitted directly
Source for sitemap discovery behavior: Google Search Central — Build and Submit a Sitemap. For full sitemap syntax, see SEORAF’s XML sitemap guide.
Step 8: Check Canonical Conflicts Affecting Crawl Signals
Canonical tags are a consolidation signal, not an access control. A URL disallowed in robots.txt can never be crawled to confirm its own canonical declaration, which undermines the signal entirely — and a canonical pointing to a noindexed URL creates an ambiguous signal worth resolving during an audit.
Step 9: Test JavaScript Rendering
Crawlers fetch raw HTML first, then attempt to render JavaScript separately. Content or directives that only exist post-render introduce a dependency on that rendering step succeeding.
| Rendering approach | Crawlability risk |
|---|---|
| Server-side rendering / static HTML | Lowest |
| Hybrid / pre-rendered | Low–moderate |
| Fully client-side rendering (CSR) | Highest |
For JavaScript SEO fundamentals beyond this audit-specific check, see SEORAF’s JavaScript SEO guide.
Step 10: Identify Crawl Traps and Assess Crawl Budget
| Trap type | How it forms | Fix |
|---|---|---|
| Faceted/filter navigation | Every filter combination generates a distinct URL | Canonicalize to the unfiltered version, or block low-value combinations |
| Session/tracking parameters | Identical content at a different URL per session | Strip parameters server-side or canonicalize |
| Infinite pagination/calendars | Sequential pages with no logical end | Cap the crawlable range or noindex beyond a reasonable depth |
| Internal search results | Every query generates a unique, thin URL | Block internal search results from crawling |
Illustrative example: a product listing page with color, size, and availability filters can generate hundreds of URL combinations from what is, content-wise, really one page.
When crawl budget actually matters: when the site is large or has an expansive URL space; when evidence shows crawl activity concentrated on low-value patterns; when important pages show delayed discovery or re-crawl; or after rapid URL-space growth without matching crawl attention. Per Google’s own crawl-budget guidance, this is mainly a concern for larger or rapidly-changing sites, not a routine task for every domain. Source: Google Search Central — Large Site Owner’s Guide to Managing Crawl Budget.
Step 11: Check Server, Access, and Authentication Issues
- DNS resolves correctly and consistently
- No password/login wall in front of content intended to be publicly crawlable
- Server doesn’t degrade specifically under bot request load
- Response isn’t served conditionally in a way that would differ from a search engine bot
Bot-management/WAF/CDN check: test directly by fetching as the specific bot user-agent and comparing against a standard browser fetch — bot-management defaults vary by provider and a change made for unrelated security reasons can quietly affect legitimate crawler access.
Advanced Diagnosis: Triangulating Crawler Simulation, Search Console, and Log Files
Every check so far relies on a simulation or a search engine’s own reporting. Neither is a direct record of what actually happened.
| Data source | What it shows | What it can’t show |
|---|---|---|
| Crawler simulation tool | What a bot could theoretically access | Whether real bots actually behave that way |
| Google Search Console | What Google reports it has crawled and indexed | Real-time activity, or activity from other bots |
| Server log files | What bots actually requested, when, and how often | Intent — logs show behavior, not reasoning |
A crawler simulation might show a page as fully accessible while log data shows the relevant bot hasn’t requested it recently — that discrepancy points to a discovery or prioritization problem rather than an access problem.
Scoped Extension: Auditing AI Crawler Access
The same access-then-discovery-then-verification logic applies to AI crawlers — with one key difference: permission is granted independently per bot.
| Signal | Traditional search bot check | AI crawler check |
|---|---|---|
| robots.txt | Googlebot/Bingbot blocks | GPTBot, ClaudeBot, PerplexityBot, Google-Extended, checked separately |
| Server/WAF | Fetch-test as Googlebot | Fetch-test as each AI crawler’s specific user-agent |
| Discovery | Standard link-following and sitemaps | Same mechanisms generally apply |
| Documentation | Google Search Central | Each vendor’s own crawler documentation, evaluated separately |
Sources for current AI crawler behavior: Google’s Common Crawlers, Anthropic’s crawler documentation, OpenAI’s bot documentation, and Perplexity’s robots.txt guidance. For the conceptual framing of AI crawler governance, see SEORAF’s crawlability overview.
Prioritizing Findings
Fixing Common Crawlability Issues
| Issue | Fix | Where to verify |
|---|---|---|
| Overly broad robots.txt Disallow | Narrow the pattern to the intended path | Re-fetch as the relevant bot |
| Unintended noindex | Remove or correct the directive at its source | Re-fetch; check Search Console status |
| Redirect chain | Point the original reference directly at the final destination | Re-fetch; confirm single-hop |
| Redirect loop | Correct the rule causing the cycle | Re-fetch; confirm termination |
| Orphan URL | Add internal links from relevant crawlable pages | Re-run crawl simulation |
| Inaccurate sitemap entries | Correct to reflect live, canonical, indexable URLs only | Re-validate entries |
| Canonical conflict | Correct the target, or merge/redirect duplicates | Re-fetch; confirm resolution |
| JS-dependent content gap | Move critical content out of render-dependency | Re-compare raw vs. rendered output |
| Crawl trap | Apply parameter handling, canonicalization, or selective blocking | Confirm activity decreases |
| Selective bot-blocking | Adjust bot-management rules for legitimate crawlers | Re-fetch as the affected user-agent |
| Blocked critical resource | Unblock required CSS/JS | Re-render and compare |
Validating a Fix
- Re-test the specific mechanism that was broken
- Confirm re-crawl activity via logs, if available
- Confirm the status change in Search Console
- Allow a reasonable observation window — timing varies by site rather than a fixed schedule
Building a Recurring Monitoring Routine
Monitor regularly: Search Console’s Coverage report, crawl stats trends, and any newly launched section against the same checks above. Run a full audit periodically, or when triggered: a major migration, a significant traffic/indexing drop, a platform change, or a new large content section.
Conclusion
Most crawlability problems aren’t mysterious once you know where to look — they’re missed because the check happens out of order, or against a single data source that doesn’t tell the whole story. The crawlability overview covers what each signal is; this article is the process for finding, prioritizing, fixing, and confirming problems on a live site — access checked before signals, evidence triangulated across multiple sources, and every fix validated against real crawler behavior rather than assumed from a corrected setting.
Use the checklist below as a working document for your next audit.