Crawlability
Crawlability is whether search engine bots can discover and request URLs on your site. It is the first requirement of technical SEO: if a page cannot be crawled, it cannot be indexed or ranked.
After this lesson you can audit and optimize how Google discovers your pages through internal links, crawl depth, crawl budget, broken link detection, orphan page resolution, and sitemap support.
This lesson covers the seven crawlability areas (leaves 3.1.1–3.1.7): internal link discovery, crawl depth analysis, crawl budget optimization, broken link detection, orphan page detection, navigation crawlability, and XML sitemap discovery support.
Why This Matters
- Crawlability is a prerequisite for everything else in SEO. A page must be discoverable before it can appear in search results.
- Google allocates a crawl budget per site. Optimizing which URLs Google crawls (and how often) ensures important content gets crawled and low-value content does not consume resources.
- Crawlability issues (orphan pages, broken links, deep depth) silently limit a site's organic potential.
Internal Link Discovery
Google discovers most new URLs through links from already-known URLs. Internal links are the primary channel for discovery.
How Google discovers URLs:
- Starts with a seed set of known URLs (from previous crawls, sitemaps, or discovered during indexing).
- Follows links from those URLs to discover new URLs.
- The process repeats, creating a crawl graph of your site.
Internal link discovery principles:
| Principle | Implementation |
|---|---|
Use HTML <a> tags with href attributes | JavaScript-based navigation events may not be discoverable by all crawlers |
| Use descriptive anchor text | Helps Google understand the target page context before crawling it |
| Link from authoritative pages | Pages that Google crawls frequently pass discovery value to linked pages |
| Avoid nofollow on internal links (unless necessary) | Nofollow links do not pass discovery signals for indexing purposes |
| Ensure links are visible (not hidden behind login, forms, or JS interactions) | Content behind user interaction may not be discovered |
Discovery check: Use the URL Inspection tool in GSC: enter a new URL and check if Google can find it via links. If not, optimize internal linking to that page.
Crawl Depth Analysis
Crawl depth measures how many clicks a page is from a crawl entry point (typically the homepage or top-level navigation).
Crawl depth levels:
| Depth | Accessibility | Example |
|---|---|---|
| Level 1 (Homepage) | Homepage + directly linked pages | Homepage, top nav links |
| Level 2 | Pages linked from Level 1 pages | Category pages, section hubs |
| Level 3 | Pages linked from Level 2 pages | Product pages, subcategory pages |
| Level 4+ | Pages linked deep within the site | Filtered pages, paginated deep pages |
Crawl depth best practices:
- Important pages should be within 3 clicks of the homepage.
- Pages at depth 4+ receive significantly less crawl attention.
- Use breadcrumb links to maintain crawl paths to deeper pages.
- Use contextual links from relevant hub pages to maintain crawl paths.
Crawl depth analysis tools:
- Screaming Frog/ Sitebulb: crawl depth report shows click distance from the start URL.
- GSC: open URL Inspection for a page — check the "Discovered but not crawled" or "Crawled but not indexed" status.
Crawl Budget Optimization
Crawl budget is the number of URLs Googlebot can and will crawl on your site within a given time frame. Optimizing it ensures Google spends crawl resources on your most important pages.
Factors affecting crawl budget:
| Factor | Impact |
|---|---|
| Site size | Larger sites may have more crawl budget but also more low-value pages consuming it |
| URL structure | Parameter-heavy URLs create infinite crawl paths; reduce where possible |
| Server response time | Slow responses reduce how many URLs Googlebot can crawl per crawl session |
| Content freshness | Google recrawls frequently updated pages more often |
| Crawl demand | High-authority pages with frequent external links may be crawled more often |
| Crawl error rate | High 4xx/5xx rates reduce crawl budget allocation |
Crawl budget optimization tactics:
| Tactic | Effect |
|---|---|
| Remove or noindex thin/low-value pages | Prevents Google from discovering and crawling URLs that do not need indexing |
| Consolidate duplicate URLs | Reduces the number of distinct URLs Google needs to crawl |
| Use canonical tags correctly | Directs Google to the preferred URL, reducing crawl duplication |
| Optimize server response time | Faster responses allow more URLs to be crawled per session |
| Block unimportant parameter URLs in robots.txt | Prevents crawl of infinite parameter combinations |
| Use nofollow on unimportant internal links | Does not pass discovery signals (but does not prevent crawling either) |
Crawl budget check: Review GSC Crawl Stats report. If Google is crawling many low-value URLs (filtered pages, paginated deep pages, parameter combinations), consolidation is needed.
Broken Link Detection
Broken links (4xx responses) create crawl path breaks. Googlebot follows a broken link and stops discovering pages beyond it.
Types of broken links:
| Type | Status Code | Impact |
|---|---|---|
| Hard 404 | 404 Not Found | Link equity loss, crawl path break |
| Soft 404 | 200 with 404-like content | Confuses Google; may be treated as low-quality |
| 410 Gone | 410 Gone | Clear signal that content is intentionally gone |
| 500 Internal Server Error | 5xx | Crawl interruption |
| Redirect loop | 302 → 302 → ... → 302 | Googlebot gets stuck, stops following |
Broken link detection tools:
- Screaming Frog: crawl and filter by status codes (4xx, 5xx).
- Sitebulb: built-in broken link reports.
- GSC: the Indexing report shows 404 pages (page-level) and Crawl errors shows server errors.
- Ahrefs/Semrush: site audit tools include broken link checks.
Broken link resolution priority:
| Priority | Issue | Response |
|---|---|---|
| P0 | 404 internal links on important pages | Redirect or fix the link within 24 hours |
| P1 | 404 internal links on any page | Fix within 1 week |
| P2 | Soft 404 pages | Improve content or return appropriate status code |
| P3 | External broken links (outgoing) | Update or remove within 30 days |
Orphan Page Detection
Orphan pages are pages that have no internal links pointing to them from any crawlable page on the site. Google may discover them through sitemaps or external links, but they lack the internal link equity and discovery support that linked pages receive.
Why orphan pages are problematic:
- Google may not discover them at all (if not in sitemap or externally linked).
- If discovered, they may not be prioritized for crawling because they lack internal link signals.
- They cannot pass link equity to other pages.
- They are invisible to users navigating the site.
Orphan page detection methods:
| Method | How It Works |
|---|---|
| Screaming Frog comparison | Run two crawls: one starting from the homepage, one from a sitemap URL list. Pages in the sitemap crawl but not the homepage crawl are potential orphans. |
| Google Analytics | Export all landing pages with sessions. Cross-reference with crawl data. Pages with traffic but no crawl path are orphans. |
| GSC Indexing report | Pages that Google has indexed but that do not appear in your crawl may be orphans. |
| Sitebulb | Built-in orphan page detection using link graph analysis. |
Orphan page fixes:
- Add contextual internal links from relevant hub pages.
- Add navigation links if the page is important for discovery.
- Add the page to the sitemap (but this does not replace internal linking).
- If the page should not be indexed, consider whether it needs to exist.
Navigation Crawlability
Navigation is the primary path Googlebot uses to discover most of your site. Navigation must be crawlable to support discovery.
Crawlable navigation requirements:
| Requirement | Implementation |
|---|---|
HTML <a> tags with href | <a href="/page-url">Link Text</a> — not button onclick, not JS pushState only |
| No JS-only dropdowns | Dropdown menus that require JavaScript events to expand may not reveal links to all crawlers |
| Logical DOM order | CSS ordering should not hide important links from the source order |
| No login walls on navigation | Navigation must be accessible without authentication |
| Breadcrumb links | Breadcrumbs provide crawl paths to deeper pages |
Navigation testing:
- Use the URL Inspection Tool in GSC: submit a URL from the navigation and check if Google can follow a path from the homepage.
- Use Screaming Frog with JavaScript rendering enabled and disabled — compare discovered links between the two crawls to find JS-only links.
- View rendered HTML from GSC URL Inspection: confirm navigation links are present in the rendered output.
XML Sitemap Discovery Support
XML sitemaps support crawl discovery by providing Google with a machine-readable list of URLs you want indexed.
Sitemap role in crawlability:
- Sitemaps serve as a discovery supplement, not a replacement for internal linking.
- Google uses sitemaps to discover URLs it may not find through links.
- Sitemaps also provide metadata: last modified date, change frequency, priority (though Google largely ignores priority).
Sitemap best practices for crawlability:
| Practice | Reason |
|---|---|
| Include only canonical, indexable URLs | Non-canonical or noindex URLs waste crawl budget |
| Keep sitemaps under 50 MB (uncompressed) or 50,000 URLs | Google's limits; split into a sitemap index if needed |
Use <lastmod> tags accurately | Helps Google decide which URLs to recrawl |
| Reference sitemaps in robots.txt | Ensures Google discovers the sitemap location |
| Do not use sitemaps as a crawl frequency tool | Sitemap submission does not force Google to crawl more frequently |
| Submit sitemap in GSC | Confirms Google has processed the sitemap and shows submission results |
Sitemap validation:
- GSC: Sitemaps report shows submitted URLs, indexed URLs, and errors.
- Check that your sitemap only includes URLs with
200status andindex, followmeta robots. - Remove URLs that return 4xx or 5xx or have noindex tags from the sitemap.
Workflow
- Crawl your site: Use Screaming Frog or Sitebulb from the homepage to map all discovered URLs.
- Analyze crawl depth: Identify important pages at depth 4+.
- Check for broken links: Fix 4xx and 5xx internal links.
- Find orphan pages: Cross-reference sitemap URLs with crawl-discovered URLs.
- Review navigation: Confirm all navigation uses HTML
<a>tags. - Validate sitemap: Submit in GSC and fix any errors.
- Review crawl stats: Check that Google is discovering new content at an acceptable rate.
Common Mistakes
Relying on sitemaps instead of internal links: Sitemaps are a supplement, not a replacement. Internal links are the primary discovery channel.
- Blocking CSS/JS in robots.txt: This can prevent Google from rendering the page correctly, which affects both crawlability and ranking.
- Ignoring soft 404s: A page that returns 200 but looks like a 404 confuses Google and wastes crawl budget.
- Treating orphan pages as normal: Pages with no internal links are invisible to users and under-prioritized by search engines.
- Submitting noindexed URLs in sitemaps: Submitting a page to a sitemap and then adding noindex sends confusing signals.
Checklist
- All important pages are within 3 clicks of the homepage or key hub pages.
- Internal links are HTML
<a>tags withhrefattributes. - Broken internal links are found and fixed.
- Orphan pages are identified and linked or intentionally excluded.
- Navigation is crawlable without JavaScript interaction.
- Sitemap includes only canonical, indexable URLs.
- Sitemap is submitted in GSC with no errors.
- Crawl stats show acceptable discovery and refresh crawl rates.
- No JavaScript-only navigation elements hide important links.